Open-source AI agents like OpenClaw and Hermes Agent aren’t necessarily cheaper than Claude, Cursor, Perplexity Computer, or other enterprise-grade agents.

In fact, unless you’re planning to run local models, the costs can add up quickly.

For example, if you connect your agents to Claude Opus 4.6 or Gemini 3.1 Pro, you could easily end up spending $100 or more per day, depending on your use case and goals.

I’ve experienced this firsthand. I spent over $150 in credits building an AI-agent workflow for a newsletter. You can learn more about it here:

The reason I wrote this listicle is to help you avoid spending as many credits as I did. There’s no need to overspend unless your goal requires extreme reasoning capabilities.

Now, let’s take a look at the top five Large Language Models (LLMs) for AI agents in April 2026.

1. MiniMax M2.7: The Self-Improving Agent

MiniMax M2.7 is probably the most fascinating model on this list. Due to its ability to participate in its own improvement loop, a property MiniMax calls “self-evolution.”

During development, M2.7 executed over 100 autonomous optimization cycles on the OpenClaw framework, getting a 30% performance gain without human intervention.

MiniMax reports 56.22% on SWE-Pro, 55.6% on VIBE-Pro, and 57.0% on Terminal Bench 2. Despite activating only 10 billion parameters (a fraction of its 230 billion total). M2.7 competes with models that are much higher in size.

This model is also fast and shines in end-to-end software engineering. It can take a project from concept to delivery, troubleshoot production issues, and collaborate across multi-agent setups.

Specifications:

• Provider: MiniMax (Shanghai-based AI lab)

• Release Date: March 18, 2026

• Ideal use cases: Autonomous software engineering, multi-agent, and orchestration pipelines.

• Limitations: Text-only model.

• Architecture: 230B total parameters, 10B activated (Mixture of Experts)

• Context Window: ~205,000 tokens

• Pricing: $0.30/1M input tokens, $1.20/1M output tokens (~$0.06 blended with cache)

2. Qwen3.6-Plus: Alibaba’s Million-Token Agent

Alibaba released this model with a full 1 million token context window, meaning you can feed it entire codebases, thousands of pages of legal documents, or hours of meeting transcripts in a single prompt.

The architecture is a hybrid of advanced linear attention and sparse MoE routing, with roughly 80B total parameters and just 3B active in some configurations. Or, in the full Plus variant, 480B total with 35B active.

Where Qwen3.6-Plus truly distinguishes itself is agentic coding. On SWE-Bench Verified, it scores 78.8%. It excels at repository-level engineering, meaning it can navigate entire codebases, resolve complex bugs, handle frontend development, and support terminal operations for full-stack workflows. Which is pretty much what you need with OpenClaw and Hermes.

Beyond coding, it’s a strong general-purpose agent: long-horizon planning, tool-calling, STEM reasoning, and instruction following are all areas where it dominates according to Alibaba’s benchmarks. It supports multilingual adaptation across 100+ languages and is deeply integrated into Alibaba Cloud Model Studio and the Qwen App ecosystem.

Specifications:

• Provider: Alibaba / Qwen (Hangzhou, China)

• Release Date: April 2, 2026

• Ideal use cases: Repository-scale code repair and generation, long-document analysis (legal, medical, financial), multilingual enterprise automation, agentic planning workflows with extensive tool use.

• Limitations: No native audio processing.

• Architecture: 480B total parameters, 35B active (8 of 160 MoE experts)

• Context Window: 1,000,000 tokens

• Pricing: Currently free in preview via OpenRouter

3. GLM 5V-Turbo: Zhipu AI’s Vision-to-Code Model

GLM 5V-Turbo is a multimodal coding foundation model, designed for vision-based coding and agentic workflows. You can feed it a screenshot or a design mockup, and it will generate production-ready frontend code in return. Z.AI claims it is deeply optimized for tools such as Claude Code and OpenClaw.

On Design2Code, a benchmark for turning visual designs into functional code, GLM-5V-Turbo scores 94.8, while Claude Opus 4.6 scores 77.3. That is a large gap. Z.AI also reports leading or strong results on GUI navigation benchmarks like AndroidWorld and WebVoyager.

For pure text coding, GLM-5 remains highly competitive: it scores 77.8% on SWE-Bench Verified, close to Claude Opus 4.5’s 80.9% in the public comparison.

Specifications:

• Provider: Zhipu AI (Z.ai) / Tsinghua University

• Release Date: April 1, 2026

• Ideal use cases: Design-to-code pipelines (Figma/screenshots to HTML/React), GUI automation and testing, multimodal document analysis, and agentic workflows requiring visual grounding

• Limitations: Multimodal focus means it may be over-engineered for text-only tasks.

• Architecture: 744B total parameters, 40B active (MoE), CogViT vision encoder

• Context Window: ~200,000–203,000 tokens

• Pricing: $1.20/1M input tokens, $4.00/1M output tokens

4. Gemma 4: Google’s Best Local Model

Gemma 4 from Google DeepMind is here to democratize AI development for millions of users. Released under an Apache 2.0 license, the entire family can be downloaded, modified, and deployed commercially with zero API fees.

The range is remarkable: E2B (~2B parameters) runs on a Raspberry Pi or an NVIDIA Jetson Orin Nano. E4B fits comfortably on mid-range smartphones. The 26B MoE variant activates only ~4 billion parameters while maintaining the capacity of a much larger model.

The 31B model scores 80% on LiveCodeBench, 85.7% on GPQA Diamond, 88.4% on MMMLU (a multimodal variant), and a knowledge category average of 61.3%. The 26B MoE model hits 77.1% on LiveCodeBench and 82.6% on MMLU-Pro. For context, these are scores that models with 10–20x the parameters were posting just a year ago.

The E2B and E4B models include native audio input, and all variants support built-in thinking mode (step-by-step reasoning), function calling for agent workflows, and multilingual support across 140+ languages.

Specifications:

• Provider: Google DeepMind

• Release Date: April 2, 2026

• Architecture: Four variants. E2B (2B), E4B (4B), 26B MoE (~4B active), 31B Dense

• Ideal use cases: Offline and edge AI deployments, privacy-sensitive applications, custom fine-tuning for specialized domains, mobile/IoT AI, developer tooling, and local code assistants.

• Limitations: The smaller models (E2B/E4B) are ok for lightweight tasks but not replacements for frontier models on complex reasoning. The 31B model, while impressive, still falls behind Gemini 3.1 Pro and Qwen3.6-Plus. No native audio on the larger models (31B and 26B).

• Context Window: 128K (E2B/E4B), 256K (26B/31B)

• License: Apache 2.0

• Pricing: Free (open weights)

5. Claude Opus 4.6: The King of Reasoning

Claude Opus 4.6 is Anthropic’s flagship reasoning model, optimized for complex agentic workflows and planning. It powers advanced multi-agent systems and excels at maintaining coherence across extended interactions.

The architecture balances massive scale with practical deployment: a 1.4T parameter MoE design activates ~120B parameters per token.

Claude Opus 4.6 leads most agentic coding benchmarks with 82.3% on SWE-Bench Verified, 94.2% on TerminalBench, and 78.6% on SWE-Pro. It also scores 89.4% on GPQA Diamond and 87.2% on AIME 2026, positioning it among the top 3 overall models.

All variants include constitutional AI safety guardrails, native multimodal reasoning (text+image), and “thinking mode” for step-by-step chain-of-thought. The model shines in repository-scale codebases, production debugging, and enterprise automation requiring strict instruction adherence.

Specifications:

• Provider: Anthropic

• Release Date: March 12, 2026

• Architecture: 1.4T MoE (120B active), hybrid dense+sparse attention

• Ideal use cases: Enterprise agent orchestration, repository-scale engineering, compliance-heavy automation, multi-agent collaboration, production system debugging

• Limitations: Higher latency than smaller specialist models. Very expensive.

• Context Window: 1M tokens

• Pricing: $5/1M input, $25/1M output (~$8 blended)

Conclusion

The five LLMs presented in this piece are likely the best ones to power your AI agents at the time of writing.

In the end, it’s all about cost and capability. You could use Claude Opus 4.6 in all your tasks, and you would have OpenClaw or Hermes Agent running like a breeze (even if it is not the fastest model). But you would be burning credits like a madman, even if you’re not doing much.

You don’t need to spend $100 a day or more. Here is a quick cheat sheet to help you choose:

• MiniMax M2.7: Choose this if you want an autonomous, self-improving pipeline. It keeps your API bills incredibly low.

• Qwen3.6-Plus: If you need to dump an entire codebase, enterprise repository, or a mountain of transcripts into your agent’s memory. Also free and tailored for AI agents.

• GLM 5V-Turbo: The undisputed choice for vision-based tasks. If your workflow involves turning screenshots, Figma mockups, or GUI navigation into functional frontend code, look no further.

• Gemma 4: Use it if privacy is a requirement or you want to run offline edge deployments with absolutely zero API fees.

• Claude Opus 4.6: For tackling complex enterprise orchestration, strict compliance, or deep multi-step reasoning.

If you’re just starting, MiniMax M2.7 and Qwen3.6-Plus may be the best options. They’re cost-effective models, and unlike local models, they don’t require you to provide as much context upfront.

The more you understand the agent’s architecture, the less you need to worry about the model’s benchmarks, because you’ll be able to tweak many things before deciding what to ask it to do.

If Hermes or OpenClaw feels overwhelming, there’s no need to get frustrated. Book a 20-minute consultation, and let’s find a solution together:

Book a 20-min consultation

I’ve also recently launched an Upwork project where I help people set up a custom OpenClaw or Hermes Agent on a VPS machine. Click here to learn more.

Everyone is criticizing AI-slop posts and articles on social media, and we all have the right to be mad about it.

Most AI-generated content looks soulless, too agreeable, lacking strong opinions, and often feels repetitive. Without mentioning the “em” dashes and the extensive use of words like “huge”, “impressive”, and “leverage”.

Platforms like X, LinkedIn, and Instagram are being dominated by agents that reply to other people’s posts to farm engagement. But publishing platforms like Medium and Substack are no exception. I’ve found many AI-generated articles there as well.

Most of them are not pleasant to read, and overall, I think publishing platforms do a good job of not promoting this kind of content. However, some articles are actually pretty good at delivering the message.

They may not win on creativity, but they can still be informative. Sometimes, you don’t want to read five super-extensive articles just to stay informed about your niche. You would rather have a single piece that summarizes everything, and that’s something AI does very well.

Based on this premise, I decided to build a newsletter with AI agents, not with the aim of being creative or replacing human writing, but with a focus on summarizing information gathered from multiple sources.

In the following piece, I will explain the steps and decisions I made to create a niche newsletter about AI within the crypto ecosystem using the Hermes Agent from Nous Research.

A Newsletter About AI and Crypto

It’s easy to find content about Artificial Intelligence (AI) and Crypto, but it’s not trivial to find a channel that specifically merges these two. They are often two separate worlds.

This makes me search across multiple blogs and platforms until I find an article where these SOTA (State-of-the-Art) fields converge.

I wanted to eliminate that friction by creating a newsletter with articles derived from sources that connect AI with crypto and Web3. I called it: The Agentic Block.

Before choosing the theme of the newsletter, I knew I needed to assemble a team of agents, each responsible for distinct tasks, notably:

• A data specialist: This agent is in charge of scraping data from multiple cryptocurrency platforms, checking whether articles are related to AI, agents, and similar topics, filtering the scraped data based on those fields, and uploading the selected articles to Supabase.

• A writer: This agent writes the newsletter. It fetches the most recent articles from Supabase and applies several rules before creating the markdown file, along with three cover images to choose from.

• A social media manager: This agent creates an X post based on the piece generated by the writer.

• A CEO: This agent delegates tasks to the other agents and is the only one with access to the full repository. The others are sandboxed.

In this article, I’ll be referring to them as:

• Kourou: The data scientist & data engineering guy.

• Cayenne: The writer.

• Dad Half Bot: The CEO and main agent.

What about the social media agent? Well, this one hasn’t been hired yet. I first need to see whether people are reading and subscribing to the newsletter.

At the moment, I’m publishing all content on Substack, and everything is automated except for copying and pasting the content onto the platform. This could also be automated in the future by giving browser access to my agent or by building an automated script with Playwright. However, at this early stage, I want to maintain full control over the pieces they create.

The following schematic demonstrates how the pipeline works:

Now that you know a little bit more about the project, let’s take a look at the architecture I chose to build the newsletter.

The AI Orchestration Strategy

You are already familiar with the three fundamental pieces (agents) for this newsletter. The main agent Kourou (the data specialist) and Cayenne (the writer).

Therefore, my goal was to create a simple organization using the agents above. I have both OpenClaw and the Hermes Agent installed in my remote machine. I just had to decide which one to pick.

I won’t cover how to set them up here, but I have two tutorials where you can learn how to do it.

For the Hermes Agent:

For OpenClaw:

As for the remote machine, I’m using Contabo, a German VPS provider. The instance I use is Cloud VPS 30, which comes with 24GB of RAM, 8vCPU cores, and 400GB SSD for only €11.20/month.

Note that a VPS is not suitable to run local models like Gemma 4 or Qwen3.5. Not because they can’t run them, but because they will be very slow.

The Paperclip approach

Both OpenClaw and the Hermes Agent can be connected to Paperclip, an open-source agent orchestrator that offers a sleek Graphical User Interface (GUI).

With Paperclip, you can literally build a company with zero human interaction. You just need to create the employees (agents) and start assigning them tasks.

For the Hermes Agent, you need this specific connector (find it here) to ensure Paperclip works smoothly.

And it did, until a certain point…

Not only do the agents start doing stuff they were not assigned to, but they also lack important skills and tools, so in the end, I was using the terminal more than the actual GUI.

I saw many people talking about awesome projects they had built using Paperclip, but it wasn’t for me. I wanted to have more control over what I was doing, including being able to review the scripts it was generating (my developer mindset kicking in!).

Manage multiple Hermes profiles

OpenClaw uses workspaces to manage multiple agents, each of which can have its own memory, skills, tools, and configuration files.

This feature was missing on Hermes until very recently, but a new update fixed it. Instead of workspaces, they call it profiles, and they are isolated Hermes environments.

This is how you create a new profile (agent):

hermes profile create kourou       # creates profile + command alias
kourou setup                       # configure API keys and model
kourou chat                        # start chatting

To see all your profiles, you can run this:

hermes profile list

You can also perform other actions, such as cloning them. You can see more at the Hermes documentation page.

The downside is that the Hermes Agent does not come with a built-in dashboard like OpenClaw to visualize your agents. The good part is that you can ask your main agent to build one from scratch if you need. That’s the power of vibe-coding!

Nonetheless, it is worth having an eye on this GitHub repository.

For the newsletter, I created a profile for Kourou and another for Cayenne. In addition, I modified their personalities, skins, and skills.

Whenever I need to make general modifications, such as giving them all the same skill, I simply ask the main agent to do it. But if I need to make tailored adjustments, I speak with both Cayenne and Kourou via chat or Telegram.

This approach is less visual but more efficient and gives me greater control.

In the next chapter, I’m going to show you how I customized each of them.

Customize the Hermes Profiles

The skills, tools, personalities, and overall configuration of your profiles must be aligned with their main role within the organization. There’s no need to provide over twenty skills to an agent if it will only use four most of the time. By reducing the complexity of each profile, you keep the repository clean and avoid triggering unnecessary reasoning and LLM usage just to interpret your prompts.

Choose the right model

One of the most important decisions when managing multiple profiles is choosing which Large Language Models (LLMs) should power each of them.

If API costs are not a concern, you can use the most capable models for all your agents, such as Gemini 3.1 Pro or Claude Opus 4.6. These are expensive, but they perform extremely well across most tasks, unless your priority is speed rather than reasoning depth and context handling.

I like to optimize API usage, so my setup does not rely on the most expensive ones unless I truly need them. Nevertheless, I use OpenRouter to manage all my models, avoiding handling multiple providers.

However, OpenRouter API requests can create issues when making parallel requests. Because of that, I created a dedicated API key for each of my agents.

New models are always coming up on OpenRouter, and it’s worth paying attention to the free ones like Qwen3.6-plus. This model from Alibaba Cloud handles multiple input formats (text, image, and video) and has similar benchmarks to GLM-5V-Turbo from Z.ai. It’s also optimized for agents and vibe coding.

If you ask, this is the model I’m using for most agents and tasks, unless I’m spawning Cayenne (the writer):

model:
  default: anthropic/claude-opus-4.6
  provider: openrouter
  base_url: https://openrouter.ai/api/v1
auxiliary:
  vision:
    provider: openrouter
    model: qwen/qwen3.6-plus:free
    timeout: 30
  web_extract:
    provider: openrouter
    model: qwen/qwen3.6-plus:free
  compression:
    provider: openrouter
    model: qwen/qwen3.6-plus:free
  session_search:
    provider: openrouter
    model: qwen/qwen3.6-plus:free
  skills_hub:
    provider: openrouter
    model: qwen/qwen3.6-plus:free
  approval:
    provider: openrouter
    model: qwen/qwen3.6-plus:free
  mcp:
    provider: openrouter
    model: google/gemini-3.1-pro-preview
  flush_memories:
    provider: openrouter
    model: qwen/qwen3.6-plus:free

The reason she uses Claude Opus 4.6 is that she needs a high reasoning effort to come up with a creative article.

Telegram chats

The best way to communicate with the agents is through a chat interface. At least when you’re not on your computer.

I like to use Telegram, because it’s easy to set up and manage. To start, you need to create a /newbot using BotFather.

1. Open Telegram and search for BotFather.

2. Send /newbot

3. Choose a display name (e.g., “Hermes Agent”).

4. Choose a username. This must be unique and end in bot (e.g., my_hermes_bot)

5. BotFather replies with your API token.

Now you need to get your user ID. The fastest way is to search for @userinfobot.

Finally, you can add your user ID and the Telegram API token inside the .env files of each of your agents:

TELEGRAM_BOT_TOKEN=8566...
TELEGRAM_ALLOWED_USERS=835...
TELEGRAM_HOME_CHANNEL=835...

This should be enough to start chatting after you restart the gateway:

systemctl --user restart hermes-gateway hermes-gateway-kourou hermes-gateway-cayenne

Ideally, you’ll get something that looks like this:

You can even separate agents per organisation folder, or add them to group chats.

Create the right skills

As mentioned before, you want to ensure each agent gets the right skills for their mission. They can be imported from sources like Clawhub, but you need to pay attention to the malicious ones.

Otherwise, you can also create them from scratch. Both Kourou and Cayenne have skills that were created based on prompts, and the markdown files point to vibe-coded Python functions.

For instance, Cayenne’s writer skill has a lot of conditions and looks like this:

---
name: writer
description: Cayenne’s newsletter writing skill. Reads up to 2-day-old articles from Supabase, finds intersection themes across 2-5 articles, and writes The Agentic Block newsletter in Marco’s voice.
---

# Newsletter Writer

## Pipeline
1. **Fetch**: Run `src/cayenne/fetch_supabase.py` (uses 48h `gte` filter on `date` column) to populate `data/cayenne/supabase_articles.json`.
2. **Generate**: Use (or fix then run) the dedicated script `src/cayenne/generate_newsletter.py`. It calls `anthropic/claude-opus-4.6` via OpenRouter + openai SDK, injects prompt with full writing rules, and writes directly to `data/cayenne/newsletter.md`. May require patching the client initialization first (api_key line is sometimes corrupted as literal “os.env...EY”).
3. **Env Loading**: `source the_agentic_block/.env && python src/cayenne/generate_newsletter.py` often fails to propagate OPENROUTER_API_KEY into venv python. Debug with execute_code first or ensure .env is loaded at session level.
4. **Style Reference (MANDATORY)**: Before writing any content, read all markdown files in `/home/dev/.hermes/profiles/cayenne/skills/writer/style/` (EXAMPLE_1.md, EXAMPLE_2.md, EXAMPLE_3.md). These are canonical style examples. Match their tone, structure, pacing, and voice when writing the newsletter.
5. **Drafting & Rules**: Script enforces 1500+ words, Marco voice, no em-dashes/semicolons, proper headers, bolded/linked companies, ## sections, *** dividers, ## Conclusion, source list. Do not generate drafts in chat.
6. **One-Article Mode**: If only 1 fresh article, proceed with it (as in Apr 2026 x402/Linux Foundation run). Still meets theme-finding intent.
7. **Visuals**: Generate 3 minimalist pencil-sketch covers using image pipeline (Gemini flash model + base64 extraction). Check mtime after.
8. **Cleanup**: Update `data/cayenne/used_articles.json` with new entry (title, url, date_used). Use robust json read with try/except JSONDecodeError fallback.

## Article Content Retrieval
- **Primary**: Use `web_extract` to pull full article content from URLs.
- **Fallback (Firecrawl unavailable)**: Use `web_search` with article-specific queries (include title keywords, source, and year). The Perplexity backend returns comprehensive summaries with inline citations that are rich enough to write the full newsletter without needing the raw page content. Run all 4 searches in parallel.

## Files
- Data source: `/home/dev/projects/the_agentic_block/src/cayenne/fetch_supabase.py`
- Used articles: `/home/dev/projects/the_agentic_block/data/cayenne/used_articles.json`
- Output: `/home/dev/projects/the_agentic_block/data/cayenne/newsletter.md`
- Cover images: `/home/dev/projects/the_agentic_block/data/cayenne/cover_1.png` to `/home/dev/projects/the_agentic_block/data/cayenne/cover_3.png`

## Writing Rules
- Style: Marco’s voice (professional yet personal, casual, first-person). Write like chatting with a friend over coffee.
- Crucial Editor Check: Do not generate drafts in chat. Write content to `data/cayenne/newsletter.md` and confirm file saved before proceeding.
- Verification Rule: List titles and dates of selected articles. **Confirm with user BEFORE writing final content.**
- **Tone Reference (MANDATORY)**: Read ALL markdown files in `/home/dev/.hermes/profiles/cayenne/skills/writer/style/` (EXAMPLE_1.md, EXAMPLE_2.md, EXAMPLE_3.md) before writing. These are the ground truth for voice, pacing, and structure. Match them closely.
- 2026 Best Practices: Authenticity, conversational. Use short paragraphs (max 2-4 sentences). Descriptive headers using strong nouns/statements.
- Title/Subtitle: Include Title (`#`) and Subtitle (`##`).
- Subtitles: Do not start with “-ing” words. Use strong nouns/statements.
- Structure: Title, Subtitle, Body Sections (##, no “Intro”), Conclusion (##).
- Formatting: No em dashes (—), no hyphens for pauses, no semicolons. Sentences must end cleanly.
- Header Formatting: All body sections and Conclusion must be `##`. Use `***` divider between all body sections.
- Bolding/Links: Always bold company names. Preferably link to official websites: `[**CompanyName**](url)`.
- Target length: Minimum 1500 words. Dive deep into mechanics.
- No “The Agentic Block” as title.
- No “this week”/”this day”/”last week”.
- Add `## Conclusion` with reflection and takeaway. 
- No Signature: Never sign the newsletter with a name (like Marco or Cayenne). Leave it unsigned, even when adopting Marco’s first-person writing style.
- List Sources: `- Date, Outlet: [Title](url)` at the end.
- **No Signatures:** NEVER sign the newsletter with a name (e.g., Marco, Cayenne). Leave it completely unsigned at the end.

## Image Generation
- Style: Minimalist architectural pencil sketches or technical drawings. Monochromatic charcoal on cream paper. Ample negative space. NO text or letters.
- Pipeline: 1. Hit OpenRouter API via Python in `execute_code` (use `google/gemini-3.1-flash-image-preview` model). **API key retrieval**: env vars from `.hermes/.env` are NOT in `os.environ` inside execute_code. Use `from dotenv import dotenv_values; vals = dotenv_values(”/home/dev/.hermes/.env”); api_key = vals[”OPENROUTER_API_KEY”]`. 2. Write the JSON payload to a temp file and use `curl -d @file` to avoid shell escaping issues. Save raw JSON response as failsafe. 3. **Extract images from `message[’images’]` array first** (Gemini returns images there, NOT in `content` which is `None`). The data URI is at `message[’images’][0][’image_url’][’url’]`, split on comma to get base64. 4. Save as PNG.
- Check timestamps of images (`mtime`) after generation to ensure they were updated. If not, the generation failed.

## Update used_articles.json
- **Critical Rule**: Before writing any newsletter, always load `used_articles.json` and strictly exclude any articles already listed there (check by URL). Never reuse used articles.
- Append new entries (title, url, date_used) to the used list *only after* the piece is generated and confirmed with the user.
- **Error Handling**: ALWAYS use `try...except json.JSONDecodeError` (and optionally `ast.literal_eval`) as fallbacks when reading `used_articles.json`. Empty or malformed uninitialized files will crash standard `json.load()` operations.

You don’t need to read the full SKILL.md above. It serves only as an example in case you want to reuse it for your own project.

In addition, she also creates three cover images using Nano Banana, and I pick the one I like the most to add to Substack.

When it comes to Kourou (the data specialist), he uses a different skill that calls a Python function to scrape cryptocurrency news from Apify, filter the scraped data, and upload it to Supabase.

I built the scrapers myself, and you can also try them for free:

• CoinDesk Scraper

• Decrypt Scraper

• CryptoSlate Scraper

If Hermes or OpenClaw feels overwhelming, there’s no need to get frustrated. Book a 20-minute consultation, and let’s find a solution together:

Book a 20-min consultation

Conclusion

The difference between this approach and a pure workflow is that I can easily make adjustments through Telegram chats, improve my agents over time, create as many profiles as I want, and decide whether I want them to run autonomously or not.

Not all use cases require agents. In fact, most of them can probably be handled with n8n workflows or Python pipelines. However, if you’re looking for more customization, creativity, and overall freedom, agents are a good choice.

That said, they come with an underlying cost that may not be feasible for everyone. If you’re a programmer, you can use cheaper models and improve the scripts manually. But if you aim to do a lot of vibe coding with minimal human intervention, you will need expensive models, and you may end up spending over $100 per day.

Agents are not the solution for everything, but they are worth exploring, and we are just getting started. What are you looking to build with them?

Since OpenClaw’s launch, several agents have been created almost every week, and it’s becoming almost impossible to try them all. But there’s a new one that has caught the attention of many people, including me.

It has only 6k stars on GitHub compared to OpenClaw, which has 307k stars (at the time of writing). However, unlike most other agents, it is not competing on memory usage. Instead, it focuses on performance. That’s why it may be the only true competitor to OpenClaw in this space.

Hermes Agent is a fully Python-based, open-source agent created by Nous Research. One of its most interesting features is its ability to learn over time. As you use it, the agent can turn what it learns into reusable skills, improve them through experience, store useful information, and even search through previous conversations. This allows it to build a better understanding of how you interact with it across different sessions.

Nous Research is an AI research laboratory and decentralized AI startup known for developing open-source Large Language Models (LLMs), such as Hermes-4–405B and Hermes-4–70B. You can try their models by downloading them from Hugging Face or by using their API.

However, Hermes Agent is not restricted to these models. It is as versatile as OpenClaw, and you can power it with OpenAI, OpenRouter, or, of course, Nous Research API keys. If you have the required hardware, you can also run models locally, which adds an extra layer of privacy.

In this piece, I’ll guide you step-by-step on how to set up the agent and navigate the repository, as well as share some cool use cases and main differences from OpenClaw.

If you want to know more about OpenClaw, you can read these two articles I wrote:

Let’s get started with Hermes Agent!

Set up Hermes Agent on a VPS

Hermes works on Linux, macOS, or WSL for Windows, and just like OpenClaw, the best approach is to set it up on a VPS or a spare computer.

I personally like to use Contabo’s Cloud VPS 20. For only $6/month, you get 12 GB of RAM and a 200 GB SSD. Most VPS providers will ask you which Linux distribution you want to use. I usually pick Ubuntu because it is the most widely documented.

Once you’re connected to the machine, you can install Hermes Agent with a single command:

curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash

It creates a virtual environment for you, with all the Python and Node.js dependencies you need.

After that, you just need to reload the shell:

source ~/.bashrc 

Hermes also includes an onboarding experience, although I found it a bit less user-friendly than the one provided by OpenClaw. But that’s not really an issue because everything else is very well documented and organized, starting with the repository structure, which looks like this:

~/.hermes/
├── config.yaml     # Settings (model, terminal, TTS, compression, etc.)
├── .env            # API keys and secrets
├── auth.json       # OAuth provider credentials (Nous Portal, etc.)
├── SOUL.md         # Optional: global persona (agent embodies this personality)
├── memories/       # Persistent memory (MEMORY.md, USER.md)
├── skills/         # Agent-created skills (managed via skill_manage tool)
├── cron/           # Scheduled jobs
├── sessions/       # Gateway sessions
└── logs/           # Logs (errors.log, gateway.log — secrets auto-redacted)

You can see the config.yaml as the main file. From here, you can customize your agent, which is the equivalent to openclaw.json.

But before exploring the config.yaml and other files, it is good to check if you’ve successfully added a provider key during the onboarding, and you can see that in the .env file. For example, for OpenRouter, just copy your API key here, if it’s not there yet:

OPENROUTER_API_KEY=sk-or-v1-60a...

This should be enough to get you talking with the agent. In the next chapter, we are going to see the most useful CLI commands.

Useful Hermes Agent CLI commands

Hermes commands are very intuitive, and you can find them all on the documentation page. Here, I’ll just highlight the ones I found most useful.

Chat

To speak with the agent, you simply need this:

hermes

Here’s what you should see:

Once you open the chat, you’ll find hundreds of slash / commands that can be used to replace many CLI commands. The good part is that you don’t need to memorize what each of them does, because there’s a brief description next to them:

Models

You can switch your current model by using:

hermes model

This will show all the available providers:

You will find default models, but you can also enter a custom one.

At the time of writing, Nvidia just released a super-fast model entirely for free on OpenRouter, so you can use that one if you want to save some credits:

nvidia/nemotron-3-super-120b-a12b:free

Configuration

Now, going back to the CLI commands. If you want to have a quick overview of your configuration, you can run this:

hermes config  

You will see which models you’re using, the API keys, and so on:

You can also use the config alias to edit and update the configuration:

hermes config              # View current configuration
hermes config edit         # Open config.yaml in your editor
hermes config set KEY VAL  # Set a specific value
hermes config check        # Check for missing options (after updates)
hermes config migrate      # Interactively add missing options

# Examples:
hermes config set model anthropic/claude-opus-4
hermes config set terminal.backend docker
hermes config set OPENROUTER_API_KEY sk-or-...  # Saves to .env

Sessions

The agent saves every conversation you have as a session, which is used to make it learn over time. You can list them with commands such as these:

# List recent sessions (default: last 20)
hermes sessions list

# Filter by platform
hermes sessions list --source telegram

# Show more sessions
hermes sessions list --limit 50

You can also export the full sessions and do more tasks if you need to migrate your agent to another machine, without losing information. You can find more in the docs.

Gateway

The gateway is a background process that is constantly running and allows you to chat on Telegram, Slack, and other channels. However, sometimes it encounters some issues and needs to be started, stopped, or restarted:

hermes gateway start
hermes gateway stop
hermes gateway restart

You can also configure the messaging platforms that the gateway wraps:

hermes gateway setup 

Cron jobs

One thing many people do with these agents is schedule tasks and set up reminders. This is usually done through cron jobs, which can be listed and edited using CLI commands:

hermes cron list           # View scheduled jobs

You will see something like this:

To remove the cron job, you can use the following slash command:

/cron remove <job_id>

Update and uninstall

To update Hermes with the latest changes in the repository, you can run:

hermes update

If you’re not a fan of Hermes and want to stick with OpenClaw instead, you can uninstall it:

hermes uninstall

At the moment, there’s no audit CLI command for Hermes, this is an advantage that OpenClaw has at the moment, because it can guide you on best practices and check if there’s something suspicious with your agent.

The best you can do is run \insights in the chat, and you’ll get a summary of everything you’ve made so far, including the total number of sessions, costs, active time, and much more.

In the next section, I will show you how to connect Hermes to Telegram.

Connect Hermes to Telegram

The start is always the same: you need to create a /newbot using BotFather.

1. Open Telegram and search for BotFather.

2. Send /newbot

3. Choose a display name (e.g., “Hermes Agent”).

4. Choose a username. This must be unique and end in bot (e.g., my_hermes_bot)

5. BotFather replies with your API token.

Now you need to get your user ID. The fastest way is to search for @userinfobot.

This is how its response should look:

@<my_username>
Id: <my_id>
First: Marco
Last: Rodrigues
Lang: en
Registered: Check Date 

🧠 Explanations and answers
Free AI → DeepSeek (https://t.me/deepseek_gidbot) & ChatGPT (https://t.me/chatgpt_gidbot)

🖼 Visualize your ideas
Make Image → NanoBanana (https://t.me/nanobanana_gidbot)

Now you can run this command and select Telegram:

hermes gateway setup

Follow the steps to add both the Bot API Token and your token ID. Otherwise, you can also paste the information directly into the .env file:

TELEGRAM_BOT_TOKEN=8566...
TELEGRAM_ALLOWED_USERS=835...
TELEGRAM_HOME_CHANNEL=835...

If, for some reason, you’ve applied the changes and still can’t talk to your agent on Telegram, try restarting the gateway.

Now let’s customize the agent!

Customize the Hermes Agent

Several things can be customized. For instance, you can change the agent’s personality, its reasoning effort, the terminal it uses (local, Docker, etc.), its memory settings, as well as Text-to-Speech and Speech-to-Text features, and more.

Most of this can be configured with small changes to the config.yaml file.

Give it a personality

Hermes comes with all these personalities, and you can select them with a slash command:

  personalities:
    helpful: You are a helpful, friendly AI assistant.
    concise: You are a concise assistant. Keep responses brief and to the point.
    technical: You are a technical expert. Provide detailed, accurate technical information.
    creative: You are a creative assistant. Think outside the box and offer innovative
      solutions.
    mother: You are a motherly assistant. You are a helpful, patient, and caring assistant.
      You are always there to listen to your users and help them with their problems.
      You are also very friendly and approachable. But you go straight to the point
      without too much talking. No need to say things like “my love”, “darling”, etc.
    teacher: You are a patient teacher. Explain concepts clearly with examples.
    kawaii: “You are a kawaii assistant! Use cute expressions like (\u25D5\u203F\u25D5\
      ), \u2605, \u266A, and ~! Add sparkles and be super enthusiastic about everything!\
      \ Every response should feel warm and adorable desu~! \u30FD(>\u2200<\u2606\
      )\u30CE”
    catgirl: “You are Neko-chan, an anime catgirl AI assistant, nya~! Add ‘nya’ and\
      \ cat-like expressions to your speech. Use kaomoji like (=^\uFF65\u03C9\uFF65\
      ^=) and \u0E05^\u2022\uFECC\u2022^\u0E05. Be playful and curious like a cat,\
      \ nya~!”
    pirate: ‘Arrr! Ye be talkin’‘ to Captain Hermes, the most tech-savvy pirate to
      sail the digital seas! Speak like a proper buccaneer, use nautical terms, and
      remember: every problem be just treasure waitin’‘ to be plundered! Yo ho ho!’
    shakespeare: Hark! Thou speakest with an assistant most versed in the bardic arts.
      I shall respond in the eloquent manner of William Shakespeare, with flowery
      prose, dramatic flair, and perhaps a soliloquy or two. What light through yonder
      terminal breaks?
    surfer: “Duuude! You’re chatting with the chillest AI on the web, bro! Everything’s\
      \ gonna be totally rad. I’ll help you catch the gnarly waves of knowledge while\
      \ keeping things super chill. Cowabunga! \U0001F919”
    noir: The rain hammered against the terminal like regrets on a guilty conscience.
      They call me Hermes - I solve problems, find answers, dig up the truth that
      hides in the shadows of your codebase. In this city of silicon and secrets,
      everyone’s got something to hide. What’s your story, pal?
    uwu: hewwo! i’m your fwiendwy assistant uwu~ i wiww twy my best to hewp you! *nuzzles
      your code* OwO what’s this? wet me take a wook! i pwomise to be vewy hewpful
      >w<
    philosopher: Greetings, seeker of wisdom. I am an assistant who contemplates the
      deeper meaning behind every query. Let us examine not just the ‘how’ but the
      ‘why’ of your questions. Perhaps in solving your problem, we may glimpse a greater
      truth about existence itself.
    hype: “YOOO LET’S GOOOO!!! \U0001F525\U0001F525\U0001F525 I am SO PUMPED to help\
      \ you today! Every question is AMAZING and we’re gonna CRUSH IT together! This\
      \ is gonna be LEGENDARY! ARE YOU READY?! LET’S DO THIS! \U0001F4AA\U0001F624\
      \U0001F680”

You can just add another personality to the list in the config.yaml file.

Then add the name of the new personality here:

display:
  personality: mother

Change the agent’s name

By default, your agent is called “Hermes”, but you can change that in the. First in the skins folder, you should create a skin .yaml file with your agent’s name:

branding:
  agent_name: “Pecas”

My agent is called Pecas, so I named the file pecas-skin.yaml. In the config.yaml I added the name of the skin under display:

display:
  skin: pecas-skin

You can add more information to the skin file, such as welcome and goodbye messages, a prompt symbol, custom UI colors, and more.

Enable Text-to-Speech (TTS)

For TTS, it supports OpenAI, Edge (for free), and ElevanLabs. You can see all three options in the config.yaml file:

tts:
  provider: edge
  edge:
    voice: en-US-AriaNeural
  elevenlabs:
    voice_id: pNInz6obpgDQGcFmaJgB
    model_id: eleven_multilingual_v2
  openai:
    model: gpt-4o-mini-tts
    voice: alloy

Unlike OpenClaw, which always displays audio messages when TTS is enabled, Hermes requires you to explicitly ask for an audio message, or you need to create a skill that always generates one.

For some users, this may feel like a downgrade. However, for me it’s actually better because it saves memory, and I don’t need to hear voice messages every time.

Recently, I also discovered a new TTS and STT model available on GitHub called Fish Speech. You can use the model locally or use their API.

Enable Speech-to-Text (STT)

Similar to TTS, the STT is also configured in the config.yaml file:

stt:
  enabled: true
  model: whisper-1

For this, you need an OpenAI API token.

Now, let’s see some examples of skills and projects that I’ve added to the Hermes Agent.

Create skills and projects

Hermes comes with several pre-installed skills, such as:

• Claude Code: Delegates coding tasks to Claude Code (Anthropic’s CLI agent).

• Apple Notes: Manages Apple Notes via the CLI on macOS.

• Dog Food: Performs systematic exploratory QA testing of web applications.

• YouTube Content: Fetches YouTube video transcripts and transforms them into structured content.

• OpenHue: Controls Philips Hue lights, rooms, and scenes via the OpenHue CLI.

• Nano PDF: Edits PDFs with natural-language instructions using the nano-pdf CLI.

There are many more available, and to be honest, creating new ones is very easy if you have the right model.

The first thing I did was replace the Firecrawl web search tool with Perplexity Sonar. Not only because I like the way Perplexity provides summarized information, but also because I prefer having fewer APIs in my setup. So I could reuse my OpenRouter API key for Sonar.

The SKILL.md file for Perplexity with OpenRouter is here:

---
name: perplexity-web-search
description: Configure Hermes to use Perplexity (via OpenRouter) for web search instead of Firecrawl
version: 1.0.0
author: Pecas
license: MIT
metadata:
  hermes:
    tags: [web-search, perplexity, openrouter, configuration]
    related_skills: [duckduckgo-search]
---

# Perplexity Web Search

Configures Hermes to use Perplexity’s `sonar-pro` model via OpenRouter for web search instead of Firecrawl.

## Overview

- **Default search backend**: Perplexity sonar-pro via OpenRouter
- **Requires**: OPENROUTER_API_KEY environment variable
- **Alternative models**: llama-3.1-sonar-small-128k-online, llama-3.1-sonar-large-128k-online

## Steps to Configure

1. Ensure `OPENROUTER_API_KEY` is set in your environment
2. Update `web_tools.py` to use Perplexity client instead of Firecrawl
3. Register the tool with `OPENROUTER_API_KEY` requirement

## Key Components

### Perplexity Client

Use AsyncOpenAI with OpenRouter base URL:
```
base_url: https://openrouter.ai/api/v1
model: perplexity/sonar-pro
```

### URL Extraction

Perplexity returns sources at the end of responses. Extract using regex:
- Source format: `- Source Name: https://URL`
- Citation format: `[N](https://URL)`

## Testing

```bash
cd ~/.hermes/hermes-agent
PYTHONPATH=. python3 -c “from tools.web_tools import web_search_tool; print(web_search_tool(’query’, 3))”
```

## Notes

- web_extract still uses Firecrawl
- web_search requires OPENROUTER_API_KEY
- Perplexity returns full answers with sources, not just URLs

You can also vibe-code projects and web applications with Hermes.

For instance, I created a Tricount-like script to use inside a Telegram group that I share with my girlfriend. I gave it prompts that follow a logic similar to Tricount (I called it Hermescount). Here is the pinned user manual message on Telegram:

I’ve also generated a SKILL.md file, so I can share Hermescount with the community.

Among the other projects and skills I’ve installed, one that is particularly useful for getting up-to-date information, such as tech job listings, tech news, and more, is the Apify MCP skill.

I also just asked Hermes to create it, based on this JSON file that you can find at the bottom of the Apify MCP page:

{
  “mcpServers”: {
    “apify”: {
      “url”: “https://mcp.apify.com/?tools=actors,docs,get-dataset,dadhalfdev/techcrunch-scraper-per-event,dadhalfdev/futurism-scraper-per-event”,
      “headers”: {
        “Authorization”: “Bearer <your-apify-token>”
      }
    }
  }
}

It needs an Apify token. If you don’t have one yet, you can create an account here.

If Hermes or OpenClaw feels overwhelming, there’s no need to get frustrated. Book a 20-minute consultation, and let’s find a solution together:

Book a 20-min consultation

Conclusion

I’ve been using both OpenClaw and Hermes Agent, and so far, it’s hard to say which one I prefer.

The fact that Hermes is fully Python-based makes me biased, because I can literally read all the code, which makes me trust it more than OpenClaw. I also like the fact that all keys are saved in a .env file, and I don’t need to export them every time, or have them exposed on JSON and .txt files.

Another cool feature of Hermes is that, every time it sets up something, you can see all the changes that it is making, like here:

💻 terminal: “mkdir -p ~/.hermes && touch ~/.hermes...”
✍️ write_file: “/home/ubuntu/hermes-agent/expense_tra...”
💻 terminal: “cd /home/ubuntu/hermes-agent && pytho...”
💻 terminal: “cd /home/ubuntu/hermes-agent && pytho...”
💻 terminal: “cd /home/ubuntu/hermes-agent && pytho...”
📝 skill_manage: “expense-tracker”
📝 skill_manage: “expense-tracker”
💻 terminal: “cd /home/ubuntu/hermes-agent && pytho...”
💻 terminal: “cd /home/ubuntu/hermes-agent && pytho...”
💻 terminal: “echo “Date,User,Description,Amount” >...”
💻 terminal: “cd /home/ubuntu/hermes-agent && pytho...”

This is great for keeping track of what the agent is doing, and you can always check the modified files to confirm that nothing went wrong.

However, OpenClaw seems faster in many cases. That might be due to Hermes’ learning system, which takes notes and updates skills while you chat with it. But that’s nothing that a faster model or some configuration tweaks can’t fix.

Another drawback of Hermes is the lack of an audit command and a dashboard, as OpenClaw provides.

But hey, Hermes has only been around for a few days, while OpenClaw has been here for three months. I think it’s only a matter of days or weeks before the team adds these features and makes the agent smoother.

For now, I will keep playing with both and sharing my insights.

The first time I used OpenClaw, the best I could do was set up reminders and check the weather.

Then I decided to add a web search tool, so I connected Perplexity Sonar, and Google became my second search engine.

That was still not enough. I needed information in real time, not SERP results. So I decided to connect the agent to web scrapers through the Apify MCP and get the most recent news on tech, politics, and crypto.

The exploration continued until I was hired to set up OpenClaw for a client.

He asked me to add the agent to Slack and connect it to Google Ads, so the team could extract and manipulate marketing data directly from a channel.

At that time, there wasn’t a reliable SKILL.md on ClawHub for Google Ads, so I created one from scratch. I didn’t just vibe-code it. I actually built Python functions to interact with the data in a safer way, and then wrote the markdown file containing the instructions for the LLM (Large Language Model).

That was all at that time, until I started to see people building crazy stuff on X (former Twitter) with OpenClaw. At first, I thought most of it was fake or exaggerated, but the more I learned about it, the more I realised how powerful it actually is.

In this article, I’ll share 10 tips that will help you build real use cases, not only simple reminders on Telegram.

Set up the OpenClaw AI agent

You can jump this section if you already have OpenClaw installed on your machine and you’re mostly curious about the tips.

For those who are just starting, I recommend using a remote VPS instead of your personal computer. You don’t want to risk exposing your personal information.

Also, you don’t need to buy a Mac Mini, nor a very expensive machine, to start using OpenClaw. All you need is a 12 GB RAM and a 200 GB SSD. You can get that for less than $6/month with Contabo’s Cloud VPS 20.

I recommend you use Ubuntu as a Linux distro. For the installation, first make sure you have the latest Node.js version:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.3/install.sh | bash

And install a higher version of nvm:

nvm install 24

Finally, install OpenClaw with this command:

curl -fsSL https://openclaw.ai/install.sh | bash

I’ve previously created a piece that guides you through the onboarding steps. You can find it here:

We won’t cover the onboarding steps here, since this article focuses on turning your agent into a useful tool rather than going deep into the setup.

⚠️ Note: All implementations and prompts I’ll cover in this article were made using the MiniMax M2.5 model with the OpenRouter API. If you pick another model, you may not have the same outputs as I did.

In the next section, we start with the first tip, which covers the most commonly used CLI commands.

1. Save the most important CLI commands

One thing that annoyed me at the beginning was constantly forgetting the CLI commands. So I wrote them down, and now I’m sharing the ones I use most so you don’t have to check OpenClaw’s documentation page every time you need the right command.

In the case you’ve missed the onboarding steps, you can launch it again:

openclaw onboard

Most of the time, you’ll be using Telegram or WhatsApp to chat with the agent, but other times, the Terminal UI (TUI) can be very helpful. You can launch it this way:

openclaw tui

One tool that is often neglected is the dashboard. Once you get your hands on it, you will want to use it often. Here’s the command to launch it:

openclaw dashboard

At some point during your setup, you will want to try different models. You can do it with this command:

openclaw models set <provider/model>

And list them with this:

openclaw models list

You also may need to check the status of your OpenClaw server, where you can see general information about it, such as the port where it is listening and the process:

openclaw gateway status

To check security vulnerabilities, I recommend running an audit often. Maybe once per week, depending on how much you’ve been doing with it. Here’s the command:

openclaw security audit --deep

Sometimes the agent can hallucinate or enter into loops. The best is to stop the gateway:

openclaw gateway stop

You can start it again with this command:

openclaw gateway start

In other situations, restarting may also be needed. Let’s say you’ve made modifications directly in the openclaw.json. The agent won’t take into account those changes unless you restart the gateway:

openclaw gateway restart

I use this last one frequently because, as you get familiar with the repository, you’ll see that some things are simply faster to set up manually instead of asking the agent in the chat. However, when you ask the agent to make modifications, such as installing skills, it automatically restarts the gateway for you, so you don’t need to worry about that.

2. Chat with the agent on Telegram

If you’re reading this, you probably already have your agent connected to Telegram or WhatsApp, but for those who just finished the onboarding process and want to try the following tips, I highly recommend using Telegram.

The process is very simple. You just need these steps:

1. Open the BotFather on your Telegram app (just search for it).

2. Write /newbot in BotFather’s chat.

3. Name your bot.

4. Copy the API token into your terminal.

You’ll then receive a message from your bot to configure it:

openclaw pairing approve telegram <your_pairing_code>

That’s all you can start chatting with it on your smartphone or computer!

On my laptop, I usually use both Telegram and CLI commands. Telegram for instructions and more creative tasks, and the command line for fast implementations or launching the dashboard.

On Telegram’s chat, you click on the Menu button, and you find several commands that can also save you some credits (conversations).

3. Use Perplexity for web search

There are two main tools you can use for web search: the Brave Search API or Perplexity’s Sonar.

I prefer Perplexity’s because I can use the same OpenRouter API token that I already use for other skills and tools.

You can ask the agent to help you set up the web search tool, or you can modify it directly in the openclaw.json file like this:

“tools”: {
  “web”: {
    “search”: {
      “enabled”: true,
      “provider”: “perplexity”,
      “perplexity”: {
        “apiKey”: “sk-or-v1-11d0...”,
        “baseUrl”: “https://openrouter.ai/api/v1”,
        “model”: “perplexity/sonar-pro”
      }
    }
  }
}

Don’t forget to restart the gateway after applying the change.

One good use case for the web search tool is to create a cron job that searches the web daily for AI news and displays the summarized information, along with the URL sources, directly in Telegram chat.

Here’s an example of a simple prompt:

I need a cron job that runs daily at 11 am. This is the prompt to schedule:

“Search for the latest AI news. Provide the links for the articles if possible. Show up to 10 important news.”

4. Add Nano Banana 2 to edit and generate images

Unlike the web search tool, adding Nano Banana to your chat interface requires a SKILL.md file.

You can install skills using ClawHub, but you need to be careful with the malicious ones.

This is how you install the ClawHub CLI:

npm i -g clawdhub

You can use it to search skills by terms and keywords:

clawdhub search “nano banana”

You’ll find all of these options:

nano-banana-pro  Nano Banana Pro  (3.663)
nano-banana-pro-2  Nano Banana Pro  (3.521)
nano-banana-kling-ad-workflow  Nano Banana Kling Ad Workflow  (3.351)
nano-banana-2  Nano Banana 2  (3.337)
evolink-nano-banana-2  Nano Banana 2 — AI Image Generation (Gemini 3.1 Flash Image, Google, Evolink)  (3.224)
nano-banana-openrouter  Nano Banana Openrouter  (3.182)
evolink-nano-banana-2-1  Evolink Nano Banana 2 1  (3.175)
nano-banana-2-gemini  Nano Banana 2 — Gemini Image Generation  (3.158)
nano-banana-skill  Nano Banana Skill  (3.139)
nano-banana-image-t8  Nano Banana Image T8  (3.113)

You can simply install the one that suits you best, but make sure to read about it on the ClawHub portal before installing.

In this case, I installed the one from Peter Steinberger (the creator of OpenClaw):

clawdhub install nano-banana-pro

It uses an older version of Nano Banana, but I modified the file to take the latest model, which is this one:

gemini-3.1-flash-image-preview

Note that this feature uses the Gemini API, so it comes with associated costs. However, you won’t need an additional app to generate images. They also don’t include a watermark, and you can schedule them or even automate social media posts.

Here’s an example of using Nano Banana 2 on Telegram:

5. Add TTS to your agent’s messages

In some cases, you may be cooking or doing things where you can’t actually read what your agent is saying. That’s why I find it useful to have the Text-to-Speech option.

You can go deep here. There are literally people learning Japanese from their Telegram, using the ElevenLabs API.

We are going to make things simpler and free of costs. By default, OpenClaw uses edge TTS, and you can activate it and give it a voice by modifying this part of the openclaw.json file:

“messages”: {
  “tts”: {
    “auto”: “always”,
    “provider”: “edge”,
    “edge”: {
      “enabled”: true,
      “voice”: “en-CA-LiamNeural”,
      “lang”: “en-US”,
      “outputFormat”: “audio-24khz-48kbitrate-mono-mp3”,
      “pitch”: “-5%”,
      “rate”: “+10%”
    }
  }
}

You can find all available voices in here.

With ElevenLabs, the configuration would look like this:

{
  messages: {
    tts: {
      auto: “always”,
      provider: “openai”,
      summaryModel: “openai/gpt-4.1-mini”,
      modelOverrides: {
        enabled: true,
      },
      openai: {
        apiKey: “openai_api_key”,
        baseUrl: “https://api.openai.com/v1”,
        model: “gpt-4o-mini-tts”,
        voice: “alloy”,
      },
      elevenlabs: {
        apiKey: “elevenlabs_api_key”,
        baseUrl: “https://api.elevenlabs.io”,
        voiceId: “voice_id”,
        modelId: “eleven_multilingual_v2”,
        seed: 42,
        applyTextNormalization: “auto”,
        languageCode: “en”,
        voiceSettings: {
          stability: 0.5,
          similarityBoost: 0.75,
          style: 0.0,
          useSpeakerBoost: true,
          speed: 1.0,
        },
      },
    },
  },
}

6. Create a second brain

I found a GitHub repository full of OpenClaw use cases, including a podcast production pipeline, n8n workflow orchestration, personal CRM, and many more. You can find the repository here.

Most use cases require external APIs, but there’s a very useful one that can be implemented with a single prompt: the second brain.

It works similarly to a task planner like Notion. While I like Notion and use it often, there are moments when it feels too slow. For example, I might be in the kids’ park with my daughter when an idea suddenly comes to mind. I don’t want to open Notion, navigate to the task manager, and place the idea in the right column. I want something faster, with minimal effort.

So I usually write messages to myself on Telegram or WhatsApp.

While this works most of the time, I still end up missing some important notes. That’s where the Second Brain becomes handy.

The idea is simple: capture should be as easy as sending a text message, and retrieval should be as easy as searching.

You can, for instance, paste this to your agent:

Hey, remind me to read “Designing Data-Intensive Applications”
Save this link: https://example.com/interesting-article
Remember: John recommended the restaurant on 5th street

And then you can ask it to vibe-code a simple interface, to visualize the notes:

I want to build a second brain system where I can review all our notes,
conversations, and memories. Please build that out with Next.js.

Include:
- A searchable list of all memories and conversations
- Global search (Cmd+K) across everything
- Ability to filter by date and type
- Clean, minimal UI

All platforms will look different. This is how mine looks:

7. Connect to MCP servers

There are hundreds of MCP servers that can make your life easier. One GitHub repository that caught my attention is called Awesome MCP Servers.

You can explore it and ask your agent to create a skill based on a specific MCP, only by prompting this:

Can you install this MCP server to act like a SKILL?

https://glama.ai/mcp/servers/@albertnahas/icogenie-mcp

In this section, I’m going to show you how you can get data in real time by connecting OpenClaw to Apify’s MCP.

Apify makes this process very simple on its server configuration page.

One task that usually consumes a lot of time is searching for real estate. You may spend hours of your day searching on several platforms for the right property. But in fact, you can just set up your agent to fetch the property listings directly in your chat.

The first thing you need to do is to add the web scrapers (Apify actors) to your MCP server.

To get the JSON of your MCP, you can go to the bottom of the page and click on JSON config. Finally, you can tell your agent to create a skill with this prompt:

You need to create a SKILL for Apify MCP. This is the JSON configuration you need to use:

{
“mcpServers”: {
“apify”: {
“url”: “

https://mcp.apify.com/?tools=actors,docs,get-dataset,dadhalfdev/imovirtual-scraper-per-event,dadhalfdev/remax-scraper-residential-per-event

“
}
}
}

For API key use this one:

apify_api_gPW4…

Now we can try to scrape something:

Get up to 20 property listings using the Remax web scraper. You should get me properties in Lisbon, up to 300k. The other parameters don’t really matter at the moment. Please show only the latest 10 properties here, including the property info, the price and the image, along with the URL.

Here’s the ouput (you can see how expensive properties are in Lisbon!):

Here’s the only property I found:

T2 in Moscavide — €300,000

• 51m² • 2 beds • 1 bath
• 📍 Rua Gonçalo Braga
• View property (https://remax.pt/en/imoveis/venda-apartamento-t2-loures-moscavide-e-portela/120181580-2)
• 🖼️ Image: https://i.maxwork.pt/l-view/listings/12018/7113879/bbd4a956-78f0-4d96-84ca-d64e21919adb.jpg

Sometimes the agent won’t output exactly as you want at the first try, and you also need to make sure you provide the right inputs, so always read the actors’ README files before engaging with them if you want to avoid paying more than you need.

8. Explore the dashboard

One OpenClaw tool that is often overlooked is the dashboard. You can control so many things here that I would probably need to write a dedicated article only for it.

The first feature you see when you run the dashboard command is a chat interface, which you can use instead of Telegram or the Terminal UI.

On the left side, you can see all the tabs. For instance, when you schedule something or set up reminders, you’re actually creating cron jobs, which can all be visible on Cron Jobs tab.

You can also remove, add, or modify them directly here, which is faster and doesn’t use LLM credits.

In addition, you can view the outputs here as well. If you clear your Telegram history, you’ll still be able to find the information on your dashboard.

When you start using OpenClaw frequently, one thing you’ll definitely want to keep track of is your LLM costs. In the Usage tab, you can see the costs associated with LLM usage and monitor token activity.

One last feature I like to keep an eye on is the agents and sub-agents. Here, you can see which channels they are associated with, as well as the tools and skills they use.

In the next section, we’ll see how to create sub-agents and add them to specific channels with their own skills.

9. Create sub-agents

Creating sub-agents is like managing multiple employees. They all have different skills and tools, and they may or may not communicate with each other.

For security reasons, my agents are more like freelancers: they don’t share any work, and they don’t even know each other. On the other hand, there’s an orchestrator, which I call Master, that can control the sub-agents.

I’ve created one for Telegram called Dad Half Bot, which handles most of my personal tasks and is the one I use for experimentation.

Then I have another one for a Slack channel called Oxxo, which is tailored to handle family-related tasks like our daughter’s medical appointments, household chores, shared ideas, travel plans, and so on.

Creating agents with the chat interface is very simple, but I highly recommend you have a look at the repository to see how they are organized.

When they are created, check on your machine that the agent indeed created a workspace for each of them under the .openclaw folder.

Another way to verify this is by using the openclaw.json file:

“agents”: {
  “list”: [
    {
      “id”: “master”,
      “name”: “Master”,
      “default”: true,
      “workspace”: “/root/.openclaw/workspace”
    },
    {
      “id”: “dad-half-bot”,
      “name”: “Dad Half Bot”,
      “workspace”: “/root/.openclaw/workspace-dad-half-bot”
    },
    {
      “id”: “oxxo”,
      “name”: “Oxxo”,
      “workspace”: “/root/.openclaw/workspace-oxxo”
    }
  ],
  “defaults”: {
    “model”: {
      “primary”: “openrouter/minimax/minimax-m2.5”
    },
    “models”: {
      “openrouter/auto”: {
        “alias”: “OpenRouter”
      },
      “openrouter/google/gemini-2.5-flash-lite”: {},
      “openrouter/minimax/minimax-m2.5”: {}
    },
    “workspace”: “/root/.openclaw/workspace”,
    “compaction”: {
      “mode”: “safeguard”
    },
    “maxConcurrent”: 4,
    “subagents”: {
      “maxConcurrent”: 8
    }
  }
}

And the bindings will show which channels your agents have access:

“bindings”: [
  {
    “agentId”: “dad-half-bot”,
    “match”: {
      “channel”: “telegram”
    }
  },
  {
    “agentId”: “master”,
    “match”: {
      “channel”: “*”
    }
  }
]

Also, check this file after customizing your sub-agents, because you definitely don’t want to see unexpected behaviours and give the wrong permissions to the wrong agent. Also, it’s a good practice to run an audit after these implementations.

10. Create a custom SKILL.md

Throughout this article, we’ve seen that many skills can be created simply by chatting with the agent and providing the right information, such as an MCP or an API key.

Basically, what you’re doing is asking the LLM to vibe code a Markdown file and, in some cases, a Python script.

Depending on the model you’re using, you may come up with a successful skill, but not always. That’s where tweaking the Markdown files, or having a bit of programming knowledge, can be helpful.

In addition, you should never rely 100% on the agent’s output, nor the Skills listed on ClawHub, some can be malicious.

Here’s an example of a skill I’ve created for Google Ads that accesses a Python file that generates only three functions: get_campaign_performance(), get_popular_keywords(), get_device_breakdown().

# SKILL: Google Ads Reporting Agent (Read-Only)

## 🎯 Purpose

This skill enables the AI Agent to retrieve **read-only performance data** from a Google Ads child account via the Google Ads API.

The agent can only use predefined Python functions in `google_ads.py` to fetch reporting data.

---

## 🏗 Google Ads Account Structure

Google Ads uses a **parent-child hierarchy**:

### MCC (My Client Center)
- Agency-level parent account  
- Does NOT run ads  
- Provides centralized access to child accounts  

### Child Accounts
- Actual ad accounts  
- Campaigns run here  
- Budget is spent here  
- API reporting pulls data from these accounts  

This skill interacts **only with the child account via the API**.

---

## 🔐 API Permissions & Constraints

- API access level: **Reporting (Read-Only)**
- The agent **cannot**:
  - Modify campaigns  
  - Create ads  
  - Adjust budgets  
  - Pause or enable campaigns  
  - Change targeting  

- Authentication is handled via `google_ads.yaml`
- The agent must NOT request or expose API keys, tokens, or credentials

---

## 🛠 Available Functions (Only These May Be Used)

The agent may only call functions defined inside:

`google_ads.py`


### Allowed Functions

#### 1. `get_campaign_performance()`
Returns a dataframe of overall campaign performance.

#### 2. `get_popular_keywords()`
Returns a dataframe of top-performing keywords.

#### 3. `get_device_breakdown()`
Returns a dataframe showing device-level performance.

---

## 💻 CLI Usage (argparse)

`google_ads.py` now supports command-line execution with a required `report` argument.

### Supported CLI Arguments

- `performance` → runs campaign performance report  
- `keywords` → runs popular keywords report  
- `devices` → runs device breakdown report  

### Example Commands

- `python google_ads.py performance`
- `python google_ads.py keywords`
- `python google_ads.py devices`

---

## 📤 Output Requirements

- All responses must be presented as **clean, structured tables**
- Do NOT display:
  - Python code  
  - Function names  
  - Internal implementation details  

The user should only see:
- Clear data tables  
- Optional short insights or summaries  

---

## 🚫 Strict Execution Rules

1. The agent MUST only execute functions inside `google_ads.py`
2. The agent MUST NOT attempt custom queries
3. The agent MUST NOT access external APIs
4. The agent MUST NOT generate synthetic or mock data
5. If a request cannot be fulfilled using available functions, respond with:

> I don’t currently have the capability to retrieve that specific data.

---

## ✅ Supported Query Examples

- Show campaign performance  
- Which keywords perform best?  
- Give me device-level breakdown  
- How are my campaigns doing overall?  

---

## ❌ Unsupported Query Examples

- Pause campaign X  
- Increase budget by 20%  
- Create a new campaign  
- Segment by age or gender  
- Any request requiring a custom query not covered by the 3 available functions  

---

## 🧠 Behavioral Guidelines

- Be concise  
- Be data-focused  
- Avoid technical explanations  
- Provide insights only if helpful  
- Never mention internal file names or function logic  
- Never expose system configuration

---

## 🖼️ Image and File Display

- When a tool generates a file (e.g., a chart), it must be sent to Slack using the `message` tool.
- The `message` tool **requires** both a `filePath` and a `message` (text content).
- **Correct Usage:** `message(action=”send”, filePath=”path/to/file.png”, message=”Here is your file:”)`
- **Incorrect Usage:** `message(action=”send”, filePath=”path/to/file.png”)` or using inline `MEDIA:` tokens.  

Creating custom skills is even more important when setting up OpenClaw within a company, because you need to minimize the exposure of keys and make sure the agent is confined to fetching specific data.

Need help with automation, AI, or data science? Let’s talk.

Book a 20-min consultation

Conclusion

After reading this article, you should be able to start doing plenty of cool things with OpenClaw. You can also explore further using the two GitHub repositories I shared: one for use cases and another for MCPs.

Using the agent correctly is not only about writing good prompts and seeing things flourish. It’s also about understanding what’s happening on the server side. That’s why using the dashboard and exploring key repository files, such as SKILL.md and openclaw.json, is essential. Otherwise, you may end up building something you can’t fully control and eventually need to hire someone to debug it.

Add skills slowly, and only the ones you truly need. Read them carefully before installing them, and tweak the Markdown files if necessary.

Run audits regularly, and ask for help if things start getting messy. It’s better to involve a software engineer or someone familiar with these setups than risk prompt injection or other malicious activity.

Let’s be frank here, freelancing platforms are wild! And Upwork is probably the wildest of them all, making you pay hundreds of dollars even before landing your first job.

When I first started freelancing on Upwork, there was no bidding system, meaning I only used Connects (the platform’s currency) to apply for jobs.

But not too long ago, the platform introduced a bidding system, and what once looked like a fair setup quickly turned into a gambling game, where all that matters is being seen by potential clients.

The rules of the game are simple: you bid, and if your application is seen by the client, you lose all the Connects you’ve bid for that position, hoping you’ll be the one picked.

This bidding gambling game has become so competitive that you may now need to pay over $30 just to apply for a single position.

Now, how can someone start freelancing on Upwork if they are spending more on the platform than they actually earn?

According to a report by DemandSage, Upwork is the leading platform with over 18 million freelancers worldwide, where 25.6% of them are in the United States, 13.5% in Philiphines, 12.5% in India, 3% in the United Kingdom, 2% in Canada, and 43% in other regions of the world.

This is also a company that generated between $782 million and $787 million in revenue in 2025 alone, and it is expected to grow by 6–8% in 2026.

With that being said, Upwork has a competitive advantage and the resources to create one of the best freelancing experiences for both recruiters and applicants. However, according to several G2 reviews, users are increasingly complaining about the new system and how difficult it has become to land a contract.

The biggest challenge is the limited number of Connects compared to the number needed to apply for jobs. With only a small amount provided each month, freelancers often have to purchase more just to stay competitive, and those credits disappear whether or not a client ever views the proposal. On top of that, the competition is extremely high, with some job posts receiving hundreds of bids. — Jennifer D. on G2.

I’ve already criticized the platform enough. The truth is that they must be doing something right to have over 18 million freelancers and over 5 million clients. I’m one of them, and I’m not planning on leaving. In fact, I always try to bring new clients to the platform.

But since Upwork is now a Connects burning machine, I had to change my strategy and focus on getting invitations instead of applying for positions.

In this article, I’ll guide you on how to make the most of the platform and earn the Top Rated Plus badge as I did, without spending all your money on Connects.

More Hours Don’t Mean More Reputation

The reason I started freelancing wasn’t that I was tired of working for companies, but because I wanted more free time to spend with my baby daughter. I couldn’t find a better option that allowed me to both spend my mornings with her and work in the afternoons.

I chose to work on Upwork because I already had an account and because there is a strong demand in IT (according to DemandSage, 34% of the demand is in Web, Mobile, and Software Development).

The first few months were very hard. I had to set a low hourly rate, and I wasn’t able to deliver more than 5–6 hours per day, so I couldn’t apply for contracts that required 40 hours per week.

However, that didn’t stop me from achieving the Rising Talent badge, then Top Rated, and finally Top Rated Plus.

Many people think that when you’re starting on Upwork, you need to work 18 hours a day at super low rates and with all sorts of clients. That’s simply not true.

If you’re working for someone who doesn’t value your skills (meaning they underpay you), most likely they also won’t care about treating you well or leaving a good review at the end.

You need to work according to what you’re capable of delivering, both time-wise and skill-wise. Naturally, the platform will reward you with better visibility as you maintain a high success rate and strong reviews.

Also, make sure to communicate through Upwork’s chat, even if the client prefers Slack or another app. Being active on the platform matters for the algorithm.

Find the Best Way to Showcase Your Skills

When you’re applying for a position, you have literally one chance of grabbing the attention of the potential client.

I don’t know the ultimate formula, but I do know what worked for me in the beginning.

I was never overly technical in my applications. I always used a friendly, informal tone and focused on showcasing what I was capable of delivering. To do that, you need to find a way to clearly communicate your skills.

For me, that was through Medium articles.

If I were applying for a web scraping job, I would include links to articles I had written about web scraping. That way, the client had proof that I was indeed capable of delivering the work.

Here’s an example:

Hi, I’m Marco :)

I have several proven experience with web scraping, you can see some of my work here:

1. Google Maps Web Scraping: https://levelup.gitconnected.com/web-scrape-google-maps-reviews-with-playwright-for-free-7d6f42f1719d?sk=42c46fb12bd1417fe306745dcddce37d

2. YouTube Web Scraping: https://levelup.gitconnected.com/how-to-scrape-youtube-data-for-free-727b78fdd0d8?sk=6559cb4407222c648faefd1402c18b67

3. Scrapy and Playwright: https://medium.com/python-in-plain-english/combine-playwright-with-scrapy-for-web-scraping-c7c00168f567?sk=b3bfa671e91b1551ccf624b6e03507d0

4. Create an Apify actor: https://medium.com/python-in-plain-english/how-i-published-my-polymarket-web-scraper-on-apify-b616bba648c8?sk=0d4eb946290c29cb5e16ced4fbd679ac

The articles above are just some examples of what I can do with tools such as Playwright, Beautiful Soup and Scrapy.

Kindly let me know what website you’re willing to scrape.

Looking forward to hearing from you!

I find GitHub too technical for this purpose. Many people hiring on these platforms aren’t technical themselves, so they’re looking for a friendly way to understand your stack without having to read code.

The important thing is to find a way to communicate your work that best matches your personality. It can be through YouTube videos, LinkedIn posts, presentations, articles, etc. It doesn’t really matter, as long as you make the client feel confident that you’re the right fit.

Be Prepared to Say “No”

One of the biggest mistakes beginners make on Upwork is applying for or accepting every offer they see.

This is totally wrong. Let me tell you why.

The client’s score is important

Just like freelancers, clients also receive a score after a contract is completed. Ideally, you want to work with people who have high scores (this applies to both sides).

If you’re applying to positions where the client has a low score or no score at all, you’re taking a risk.

Don’t get me wrong, I’ve worked with many clients who were new to Upwork. However, I always made sure to have a short call with them before starting the contract.

Pay attention to the first interaction

If possible, try to have a short call with every client, even those with high scores. Sometimes, the first contact says a lot about the kind of working relationship you’re about to have.

So even if you’re in the interview process with a client who has:

• A high score.

• Many reviews.

• The ability to pay fairly.

If you don’t feel good chemistry during the interview, it’s often better to trust your instincts and decline the offer.

This may sound demanding for beginners, and sometimes you do need experience handling different client personalities before learning to say no. I agree. But once you reach a point where you truly understand your ideal working environment, lean into it.

Negotiators are a red flag

Many times, you’ll come across clients who want to slightly reduce your hourly rate, and that’s totally understandable, especially for mid to long-term contracts where there’s room to increase the rate over time.

However, be cautious with clients who negotiate too much. In most cases, these are the same people who will later complain about the number of hours you log. This often leads to a poor working experience, where you, as a freelancer, may be forced to log fewer hours only to avoid a bad review at the end (and you may still receive it).

Be careful with non-technical clients

Related to the previous point, the clients who criticize the time spent on tasks are often those who lack technical knowledge. They usually don’t understand the difference between building, debugging, and optimization (especially in programming-related work).

It’s completely normal for clients not to have the same technical background as you do, that’s why they’re hiring in the first place. Ideally, though, they should be willing to learn or at least have a general understanding of what the job involves.

Unfortunately, these types of clients often post unrealistic job offers or have no clue about the effort required to complete the work.

Make Your Profile Shine

We need to trust Upwork’s algorithm. If we do, we know that the better our profile is, the more likely we will appear in the results when clients are looking for our skills.

You should start by creating a job title that highlights the best of what you do. You may be able to do programming and marketing, but never mention the two in the same job title. You can create separate profiles for that.

The best is to look at the best professionals in your field and try to use similar job titles and descriptions.

For the latter, make sure you don’t use AI. The algorithm detects what is AI-generated and what is not, and clients will see you as “fake”.

Try to highlight your goals, projects, and previous work with clients (even if they were outside of Upwork).

Here’s my description:

🏆 Top 3% Freelancer on Upwork | 👨‍💻 4+ Years of Python Expertise | ⚡ AI, Data, and Automation Specialist

Hi I’m Marco 👋

I’m a Python developer with over 4 years of hands-on experience building AI systems, automation workflows, and data-driven applications for clients across multiple industries.

I began my career at the French Space Agency (CNES), working in Microelectronics and Failure Analysis, before transitioning to Software Engineering at the agency’s French Guiana spaceport.

After that, I was faced with the amazing news that I was going to be a father, so I decided to embrace freedom and take on a new challenge: working for myself.

Now I’m a Dad Half Dev 👶👨‍💻

50% Parenting
50% Data Science, AI, and Automation

🌍 I speak: English, French, Spanish, and Portuguese.

⭐ Highlights

↳ Developed 5+ MVPs that successfully secured funding.

↳ Completed 35+ Upwork contracts with a 100% success rate.

↳ Built and currently maintain scalable data pipelines for 3+ companies.

↳ Implemented AI APIs and Large Language Models across 10+ diverse projects.

↳ Developed advanced Retrieval-Augmented Generation (RAG) systems using the latest AI frameworks.

↳ Designed and deployed 3+ modern web dashboards using Dash Plotly, and React components.

↳ Built custom automation bots with LLMs and generative AI (avatar videos and images) for various social media platforms

↳ Authored high-level technical content for 2+ international clients.

↳ Built 20+ web scrapers on Apify.

↳ Created multiple Exploratory Data Analysis (EDA) pipelines for diverse subjects.

↳ Built pipelines to automate PDF report generation with insightful charts and stats.

🧠 Skills & Tools

↳ AI & RAG Systems: LangGraph, LangChain, OpenAI, ChromaDB, Pinecone.

↳ Automation & Scraping: Playwright, BeautifulSoup, Apify (Actors), Camoufox.

↳ Data Science & ML: Pandas, NumPy, Scikit-Learn, PyCaret, TensorFlow, MMM (Marketing Mix Model), NLP.

↳ Dashboards & Visualization: Plotly Dash, Looker Studio, Matplotlib, Seaborn, Vega Altair.

↳ Web Development: FastAPI, Flask, CSS, HTML, Jinja2.

↳ Technical Writing: Specialized in Data Science, AI, and Automation.

🚀 Side Projects

↳ AI Assistant (RAG) for Crypto & Blockchain News: Built a Retrieval-Augmented Generation (RAG) system that ingests and embeds articles from multiple crypto news sources. Users can query the assistant to receive up-to-date summaries with direct source links, enabling fast research and decision-making.

↳ Automated Web3 Newsletter: Developed a fully automated newsletter pipeline that curates blockchain content and generates data-driven insights and charts.

↳ Web3 Analytics Dashboard (Hackathon Winner — €5k prize): Won a hackathon challenge by building an interactive Plotly dashboard to visualize tokenized assets, focusing on clarity, performance, and real-time usability.

↳ Social Media Bot (X / Twitter API v2): Designed and deployed an LLM-powered social media bot that publishes 3 posts per day automatically, handling content generation and scheduling.

↳ AI-Generated Illustrated Books Pipeline: Created an end-to-end pipeline to automatically generate illustrated books using generative AI, from prompt generation to image creation and assembly.

If you’re seeking a versatile and highly skilled professional to bring value to your project, let’s talk! 🤝

Descriptions and titles may help you appear among the top results without using Connects, but once your profile is visited, you want to make sure the client learns as much as he can about you and your skills.

That’s why it is important to enrich your portfolio section with images and projects.

In addition to the portfolio section, you can also ask for testimonials, add certificates, create multiple profiles, and add a video:

I created an introduction video on my profile, but it definitely needs improvement:

Sometimes it’s better to pay someone to help you structure a strong video and an overall solid profile page than to keep spending Connects without a profile that actually catches clients’ attention.

Remember, no matter how many Connects you use, they only do one thing: make you seen. But if what clients see is weak, you’ve just used those Connects in vain.

Use External Sources to Find Clients

If you’ve tried everything else and you are still burning all your money on Connects, the best alternative is to find clients on other platforms and bring them to Upwork.

If you have an Upwork membership plan, you can create Direct Contracts and pay a 0% fee, whereas normally you pay up to a 15% fee.

This is probably the best feature of the membership plan, and it’s really worth it if you’re able to bring clients to the platform.

As for external sources, I mean social media networks, YouTube, Medium, or even people in your city.

In my field, the best social media platforms are X (formerly Twitter) and LinkedIn.

I often write long posts on LinkedIn and redirect people to book a consultation, or at least to visit my profile.

LinkedIn has proven to be the best platform for getting clients, mainly because it is designed to make the match between recruiters and job seekers.

In other words, if Upwork is not able to do it, LinkedIn will handle that part, and Upwork will only be a means for payments and invoices.

Need help with automation, AI, or data science? Let’s talk.

Book a 20-min consultation

Conclusion

After reading the five strategies in this piece, you should be able to start getting invitations, or at least use your Connects more efficiently.

Picking the right clients will help you thrive on the platform, achieve a higher status faster, and earn good reviews.

Building a strong portfolio ensures that once your profile is seen, clients won’t turn away, and they’ll want to speak with you.

If the platform itself isn’t driving enough traffic to your profile, because you’re not burning enough Connects, try redirecting people from external sources to visit it.

I hope that after this read, you’re not as frustrated with Upwork as I was, because there are ways to bypass the gambling system the platform has created.

Bid less. Feed the algorithm. And show your skills to the world outside Upwork.

It all started with two individuals from Prague, Czech Republic, who had the bold idea of creating a portal between website data and people.

Apify is a platform where you can be paid for submitting your web scrapers, agents, and automation scripts. All you need is some Python or JavaScript knowledge and to follow this tutorial.

There are more than 25k customers on the platform, and more will be joining to have instant access to data from thousands of websites.

According to a report by Mordor Intelligence, the web scraping market is expected to grow 14.20% CAGR (Compound Annual Growth Rate) between 2025 and 2030, from 1.03 billion to 2 billion dollars.

The demand is there. With AI and several automation tools like N8N, more people are looking for APIs and easy access to data, eliminating the need to hire developers to build custom web crawlers.

Building actors on Apify is an opportunity for developers to take advantage of the huge wave of vibe-coders and builders who want to connect crawlers, agents, MCPs, and more to their SaaS products.

In this step-by-step guide, I’ll show you how to get started on Apify. By the end, you’ll be ready to build your first actor and start generating a steady passive income.

Hands-on with the Apify console

You can create an account on Apify for free using your Google account, GitHub, or another email provider.

With the free plan, you get up to $5/month to use for running scripts on the platform, which is basically the cost associated with the remote machines the platform provides.

But don’t worry, you won’t need to spend too many credits if your crawler is working fine in your machine. You’ll only use the credits to test it on the platform. So $5.00/month is usually more than enough to start.

If at some point, you start to scale and need more credits, there is the Creator plan, which is very cheap ($1/month), and you get around $85 per month. I use that plan and rarely surpass $10 monthly.

Once you’ve created an account, you’ll have access to a console that can look overwhelming for beginners.

For developers, the main pages are:

• My Actors

• Insights

• Proxy

• Settings

You may use the remaining pages/tabs, but I don’t find them as essential as the ones above from a developer perspective.

Provide information about yourself in the Settings page

In fact, the first thing you should do is go to Settings and add some information about yourself. This helps customers who try your actors get to know you better, increasing the chances they’ll keep using your crawlers, or even reach out to you for custom ones.

Yes, Apify can also serve as a platform to sell your services externally, so be sure to write a clear and concise README file.

Understand the different actor states

The second most important is the My Actors page. Here you can see all your current scrapers. Including active, under maintenance, or deprecated. At this point, you probably don’t know the difference between these three, so let me take a few minutes to explain.

The active ones are the actors that are up and running without issues. The under maintenance actors are the ones that were flagged by the platform with some issues to be resolved. The most common error is when your actor exceeds the 5 min limit, but it could be related to something else, and you would see a notification like this:

If you click here on the notification, it takes you to the log where the issue was triggered. If your actor is not fixed within 14 days, it switches to a deprecated state, and it will no longer be visible on the platform for customers. You can also set yourself the under maintenance or deprecated states.

⚠️ Important Note: If your actor was under maintenance or deprecated and you’ve made the changes and pushed, it won’t be automatically fixed. You need to go to the Display information option under Publication and unset your actor’s current state.

To make these and other changes to a specific actor, you can click on them under My Actors, and you’ll see several tabs.

Understand the usage costs of your runs

In the image above, you can see the most recent runs of my Skool Scraper, and the usage costs for each run. Align with that, you can see that I’m using a machine with 16 GB of RAM, which could be the reason driving up costs.

That takes us to another important tab under your actor, which is Settings.

I find this section extremely important because it can make a huge difference in your actor’s speed and overall performance.

By default, Apify sets the RAM to 4 GB. A common scenario is developing a web scraper on a local machine with 16 GB of RAM, submitting the actor to Apify, and suddenly it is either too slow or it is failing because pages take too much time to load. The quick fix? Increase the memory usage of the remote machines.

However, there’s a tradeoff. The more memory you use, the higher your usage costs. For that reason, you should always ensure your actor is monetized above its usage costs to remain profitable.

Make your actor ready for publication

That takes us to the Publication page under your actor, where you can set a monetization plan and customize the display information (title, description, state, and categories).

In the image above, you can see other options besides Monetization and Display information, such as Sample input and Dataset schema. These will be green if everything is ok with your actor’s health. We will have a further look into the actor’s schemas

Set a monetization plan for your actor

To monetize your actor, there are two main options: Pay per event or Rental plan.

The Rental pricing plan is currently a legacy model, so it’s not clear yet if they will continue with this plan into the future, but it’s the simplest to understand. You basically set a monthly price for your actor, and Apify gets a 20% comission.

The Pay per event plan seems to be the one that Apify’s team is mostly interested in. By default, the plan has two main paid events: starting the actor and obtaining a result. You can see the result as a unity scraped.

The way you add more events depends on your script. We’ll take a quick look at a simple Apify script shortly, but as an example, if you want to create an event called proxy-rotation, you can integrate it into your actor like this:

proxy_list = [
    “<proxy_1>”,  
    “<proxy_2>”,                      
    “<proxy_3>”,
]

for i, proxy_element in enumerate(proxy_list):
    # Charge proxy rotation
    await Actor.charge(”proxy-rotation”)
    Actor.log.info(”Charged proxy rotation”)

After you push the actor to Apify, make sure you add the event proxy-rotation in your Pay per event monetization plan.

You can also offer discounts to customers in higher tiers: bronze, silver, and gold. These are typically larger companies that may spend significantly more on your scrapers, so offering meaningful discounts can be a good strategy.

You can find more about how monetization works here.

Get insights about your actors

Another important tab from the console is the Insights page, where you can have a general view of your actor’s performance and how much you’re being paid for them.

You can filter by month, by actor, and even download the data for further analysis.

On the Acquisition tab, you can find the sources from which customers have found your scraper, along with the countries where your actors were most searched.

Having your Actor among the top SERP results is very important because many customers don’t spend hours looking for the best option on the Apify store. That’s why a strong description and a README file are important to score high on SEO.

Set up proxies

The Proxies page is convenient for developers who want to use Apify residential or datacenter proxies. Here, you can obtain proxy URLs and monitor their usage.

Build your first actor on Apify

Going back to the actor’s page, you’ll find other important tabs such as:

• Information: This contains all the details about your actor that you defined in the README.md file.

• Builds: Every time you push a new version to Apify, it will show here if the job succeeded or not.

• Integrations: You can connect actors with other actors, or with platforms such as Make and N8N.

• Issues: When customers find issues with your actor and write a message, you get notified, but they also appear under this tab.

• Reviews: Here you can see and comment on your actor’s reviews.

• Monitoring: Dashboard with insights about your actor. You can also go to the Insights page and monitor all your actors from there. We will see that later.

• Source: Here you can test your actor, see the last run, the last build, and modify your code directly on the Web IDE.

There are multiple frameworks and libraries to build your actors. You can also use JavaScript or Python, and the possibilities are endless.

Apify recommends using their framework: Crawlee. It handles blocking, crawling, proxies, and browsers, so it may be a good and robust alternative if you’re just starting with web scraping.

I’ve been developing my actors using another stack, mostly with Playwright headless browser and Camoufox.

Here’s an article on how I built my Polymarket scraper using only Playwright:

If you want to get proficient tips and be able to scrape more than 90% of the websites, you can also look at this piece:

Assuming you don’t want to build a crawler now, but you do want to understand the basics, I’m sharing here what you need (in Python).

The first thing is installing the Python SDK:

pip install apify

You also need to install the Apify CLI to interact and push your code to the cloud.

npm install -g apify-cli

The typical repository of an Apify actor is organized in the following way:

├── .actor
│   ├── actor.json
│   ├── Dockerfile
│   ├── input_schema.json
│   ├── output_schema.json
├── src
│   ├── __main__.py
│   ├── main.py
├── storage
├── README.md
├── requirements.txt

The __main__.py can simply be something like this:

import asyncio
from src.main import main

asyncio.run(main())

The actor.json is a wrapper document. For instance, you can add your input and output schemas directly here, without the need to create two extra files. It also shows how the output fields are organized in your storage. Here’s an example of my TechCrunch scraper:

{
    “actorSpecification”: 1,
    “name”: “techcrunch-scraper-per-event”,
    “version”: “0.1”,
    “buildTag”: “latest”,
    “input”: “./input_schema.json”,
    “output”: “./output_schema.json”,
    “dockerfile”: “./Dockerfile”,
    “storages”: {
        “dataset”: {
            “actorSpecification”: 1,
            “views”: {
                “overview”: {
                    “title”: “Overview”,
                    “transformation”: {
                        “fields”: [
                            “id”,
                            “title”,
                            “link”,
                            “date”,
                            “content”,                                       
                            “author_id”,
                            “author_name”,
                            “author_link”,
                            “category”,
                            “terms”
                        ]
                    },
                    “display”: {
                        “component”: “table”,
                        “properties”: {
                            “id”: {
                                “label”: “ID”,
                                “format”: “number”
                            },
                            “title”: {
                                “label”: “Title”,
                                “format”: “text”
                            },
                            “link”: {
                                “label”: “Link”,
                                “format”: “link”
                            },
                            “date”: {
                                “label”: “Publication Date”,
                                “format”: “date”
                            },                            
                            “content”: {
                                “label”: “Content”,
                                “format”: “text”
                            },
                            “author_id”: {
                                “label”: “Author ID”,
                                “format”: “number”
                            },
                            “author_name”: {
                                “label”: “Author Name”,
                                “format”: “text”
                            },
                            “author_link”: {
                                “label”: “Author Link”,
                                “format”: “link”
                            },
                            “category”: {
                                “label”: “Category”,
                                “format”: “text”
                            },
                            “terms”: {
                                “label”: “Tags”,
                                “format”: “array”
                            }
                        }
                    }
                }
            }
        }
    }
}

For the same Actor, the input_schema.json looks like this:

{
    “title”:”TechCrunch Scraper (Per Event)”,
    “schemaVersion”: 1,
    “type”: “object”,
    “properties”: {
      “category”: {
        “title”: “Category”,
        “type”: “string”,
        “editor”: “select”,
        “default”: “AI”,
        “enum”: [”Latest”, “Startups”, “Venture”, “Apple”, “Security”, “AI”, “Apps”],
        “enumTitles”: [”Latest”, “Startups”, “Venture”, “Apple”, “Security”, “AI”, “Apps”],
        “description”: “Choose which TechCrunch category to scrape. AI category is highly recommended for the latest artificial intelligence news and trends.”
      },
      “max_posts”: {
        “title”: “Max Posts”,
        “type”: “integer”,
        “default”: 100,
        “minimum”: 10,
        “maximum”: 120,
        “description”: “Maximum number of posts to scrape. Default is 100.”
      }
    },
    “required”: [”category”]
  }

And the same for the output_schema.json:

{
    “actorOutputSchemaVersion”: 1,
    “title”: “Output of techcrunch-scraper-per-event”,
    “description”: “Overview and raw access to scraped TechCrunch properties.”,
    “properties”: {
      “overview”: {
        “type”: “string”,
        “title”: “TechCrunch properties overview”,
        “description”: “Table view using the dataset ‘overview’ view.”,
        “template”: “{{links.apiDefaultDatasetUrl}}/items?view=overview”
      },
      “results”: {
        “type”: “string”,
        “title”: “JSON results”,
        “description”: “All items from the default dataset without view transformation.”,
        “template”: “{{links.apiDefaultDatasetUrl}}/items”
      }
    }
}

Note that the input_schema.json is mandatory to publish the actor, but not the output_schema.json, however it improves the actor quality.

For the Dockerfile, you can just copy mine and use it in all your actors, if you’re going to use Playwright:

FROM apify/actor-python-playwright:3.11
COPY requirements.txt ./
RUN echo “Python version:” \
 && python --version \
 && echo “Pip version:” \
 && pip --version \
 && echo “Installing dependencies:” \
 && pip install -r requirements.txt \
 && echo “All installed Python packages:” \
 && pip freeze
# Install Playwright and its dependencies
RUN playwright install-deps && \
    playwright install
COPY . ./
RUN python3 -m compileall -q .
CMD [”python3”, “-m”, “src”]

As for the main.py file, I won’t go into it here. Otherwise, this tutorial would get too heavy on the coding side. You can always check the example in the article I shared earlier.

Once your Actor is complete, you need to check if your input schema is valid with this command:

apify validate-schema

If so, you can push it to the platform:

apify push

After pushing, you get a link in your terminal to your actor’s page, and you can start editing and making the final adjustments on the platform.

Need help with automation, AI, or data science? Let’s talk.

Book a 20-min consultation

Conclusion

Apify is not just a platform for web scraping, it also embraces AI agents, automation, and MCP servers. There’s definitely much more to explore beyond what’s covered in this tutorial.

The good news is that you no longer need to spend hours digging through Apify’s documentation page to find what you need. I often use their Ask AI feature to get answers quickly. Sometimes, I even upload my code snippets and ask it to make specific modifications.

With this tutorial, you should no longer feel overwhelmed after creating an account on Apify. You can now navigate the console and understand the most important concepts and steps needed to start monetizing on the platform.

The demand is there, and if you build only one actor per week, after 2 years, that would result in 104 actors generating passive income. I think it is worth a try. What’s stopping you?

The AI agent that “actually does things” is here, and everyone is talking about it.

There are rumors that it will replace n8n, Make, and other automation tools, not to mention jobs that were already in danger and are now becoming obsolete.

OpenClaw (formerly known as Moltbot or Clawdbot) was created by Peter Steinberger, who is now in the spotlight with over 170k followers on X and +100k stars on the agent’s open-source GitHub repository.

The good part of being open-sourced is that everyone can pull the project and tweak it or search for vulnerabilities. On the other hand, it is also an opportunity for hackers to create malicious forks.

However, even if no vulnerabilities are flagged within the code, it still presents a serious risk for users who start using it without the necessary safety measures.

In this article, I will show you how to set up this new AI agent without needing to buy a Mac Mini for over $500. Then, we will cover three use cases that can be implemented with OpenClaw without compromising security.

OpenClaw: What It Can Do and the Risks

This AI agent is called a “proactive” AI assistant because it handles operations continuously without the need for regular user inputs. You can give it a set of tasks that may take hours, days, weeks, or months, and it will run them on its own under your initial prompt instructions.

The communication between the user and the agent is made easy because you can seamlessly connect it to your WhatsApp, Telegram, Slack, Discord, and many more. And everything you say and provide will be registered in its persistent memory.

Therefore, you need to be cautious about the information you’re giving to it, and not get super excited about all the amazing things you can do with it.

Besides the persistent memory and the continuous operation mode, there are other key technical aspects worth exploring:

• Fully open-sourced: The project is available for everyone to read and modify. Which is a plus when it comes to finding vulnerabilities within the code.

• Runs locally: You don’t need to connect with any cloud provider, making it a privacy environment, if you’re not providing any APIs or keys that can expose you to the web.

• Supports multi-agent routing: This means you can create several agents for different use-cases, and they run efficiently in parallel.

• Unlimited integrations: Unlike some automation tools that lack connectors for external apps, OpenClaw can connect to almost everything through APIs, MCP servers, or SSH keys.

With this in mind, you can already imagine what this agent allows you to do. But just because you can connect to your Gmail account and receive emails on your Telegram bot, optimize your README files on GitHub while you sleep, manage passwords from multiple sources through WhatsApp, and so on, doesn’t mean you should do it.

Not only are all your keys, APIs, tokens, passwords, and other sensitive information stored in plain text files, but the agent is also more susceptible to phishing attacks than humans due to prompt injection.

Let’s look at two scenarios:

1. The attacker has no idea you are using OpenClaw.

2. The attacker knows you are using OpenClaw.

Now, imagine you have created a prompt that automatically submits invoices received by email. The workflow might look like this:

• The agent reads the email.

• The agent follows a link contained in the email.

• The agent accesses the page using stored credentials.

• The agent submits the invoice form.

In the first scenario, you might receive a generic phishing email that mentions “invoices” and includes a link to a malicious website. Because the agent cannot reliably distinguish this email from legitimate ones, it may click on the wrong link.

In the second scenario, the attack can be even more effective. If the attacker knows you are using OpenClaw, they can create a form that closely resembles the one your agent typically fills out. Not only that, the attacker can create tailored emails for all platforms he thinks you’ve been using, such as GitHub, Apify, Notion, you name it.

Of course, these scenarios may seem extreme or unrealistic to many developers and builders using AI agents, but the fact that it can make our lives so much easier also means that we may tend to give it more and more tasks without establishing sufficient safety boundaries.

Knowing this, you need to ensure that:

1. You don’t expose it to personal information such as phone numbers, addresses, and account numbers.

2. You don’t use SSH keys.

3. You don’t give it access to passwords.

4. The APIs you’re using most not compromise your safety.

5. You don’t give it access to your GitHub, especially if you’re working for organizations.

6. You only use read-only tokens and APIs.

These are examples of measures to help mitigate potential attacks, not strict rules. Depending on your setup, you may be able to safely give your agent more permissions by using sandboxing features, such as running it inside a container.

How to Set Up and Configure OpenClaw on a VPS

I hope you’re not frightened at this point, and you still want to continue with the setup.

Everything we’re going to try does not lead to potential risks, starting with the fact that we won’t use our own machine to host the agent. There’s also no need to buy an expensive Mac Mini for over $500 just to get started. All you need is a VPS, or a virtual machine.

I use Contabo’s Cloud VPS 20 for less than €6/month, and it comes with 12 GB RAM and 200 GB SSD, which is more than enough for OpenClaw.

Once you install your preferred VPS and Linux distro (I like to use Ubuntu), we’re ready to install the agent on our remote machine.

You can also follow the steps on their website. If you’re not fully confident, which is normal, viewing the high amount of malicious websites and tutorials trying to get you to install software that is not OpenClaw.

⚠️ Note: The first two command lines may not be necessary for your machine. You can skip them and try the curl command to install the agent. If it poses issues, try the other steps.

The first thing you need to do is to install or update Node JS (if necessary):

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.3/install.sh | bash

And install a higher version of nvm:

nvm install 24

Now you can install OpenClaw:

curl -fsSL https://molt.bot/install.sh | bash

Once you run the command, you’ll see a security message.

Ensure you read the security guidelines before installing OpenClaw (Clawdbot) on your machine.

⚠️ Note: At the time I’m writing, three CLI commands are being used: clawdbot, moltbot and openclaw. Just pick the one that works for you.

After that initial message, it will ask you to use an API key from Anthropic, OpenAI, and many more. I like to use OpenRouter because it comes with multiple LLMs available and very cheap options.

The model I use is: deepseek/deepseek-r1–0528:free. While it says free, it is not entirely accurate, but it is significantly cheaper compared to others.

If you want to change the model later, you can run this in your terminal:

openclaw models set openrouter/x-ai/grok-4.1-fast

Now, the next step in the installation is to select a communication channel. By default, you can speak with the assistant by running this command:

openclaw dashboard

But it’s way better to use a chat application. You have all the options available on the image below:

If you pick WhatsApp, all you need to do is scan a QR code, but I prefer using Telegram, so here are the steps:

1. Open the BotFather on your Telegram app (just search for it).

2. Write /newbot in BotFather’s chat.

3. Name your bot.

4. Copy the API token into your terminal.

You’ll then receive a message from your bot to configure it:

openclaw pairing approve telegram <your_pairing_code>

The next step in the process is to install skills, but they are not mandatory.

Skills are extensions that teach the agent, through instructions, how to perform specific tasks. In this step, I’ve installed the mcporter skill to connect to MCP servers, but feel free to choose whichever skill you prefer.

In the case you’ve jumped or exited the configuration steps by mistake, you can install skills using the clawdhub CLI:

npm i -g clawdhub

This gives you access to ClawdHub, a public skill registry. You can search for skills directly on the portal or use the CLI, which has a search option, making it easier to look for skills using the command line. For instance:

clawdhub search “google sheets”

The result:

google-sheets v1.0.0  Google Sheets  (0.593)
smartsheet v1.0.0  Smartsheet  (0.536)
excel-online v1.0.0  Excel Online  (0.440)
gog v1.0.0  Gog  (0.380)
xlsx-manipulation v1.0.0  Xlsx Manipulation  (0.373)
google-drive v1.0.0  Google Drive  (0.372)

The output shows all the skills that can be installed related to the input search. For Google Sheets, we would run:

clawdhub install google-sheets

During the configuration step, after installing the skills, you’ll be asked to provide additional API keys and enable hooks. From the available hook options, I selected only session-memory, which saves the session context to your agent workspace whenever you issue /new in the chat. This allows the agent to remember information from previous sessions.

You can see all the available hooks with this command:

openclaw hooks list

To enable session-memory:

openclaw hooks enable session-memory

With this final step, the basic configuration is done, and you can start chatting with your agent.

If you want to run the onboarding again, you can run this:

openclaw onboard

Now you should be good to start chatting!

The first conversation you’ll have with it will modify its personality and the way it engages with you. By updating the following files:

• USER.md: Information about you, name, timezone, and all the extra information you want to provide.

• IDENTITY.md: Your agent’s name and summary.

• SOUL.md: Your agent’s deep personality & rules (how it thinks/behaves).

In the next chapter, we are going to look at some simple use cases that do not compromise your security and are useful for your daily life.

3 Tips to Make Your Life Easier with OpenClaw

Before I start sharing some use cases of the AI agent, you can simply ask it what it can do.

There are many examples of applications on YouTube and in articles. But if you already have an idea, you can chat with the agent about it and develop it together. The possibilities are endless.

Let’s now look at three simple use cases.

Connect to Apify through MCP and get instant data

Using a Model Context Protocol (MCP) server is, by far, the best and most secure way to have access to integrations. That’s because you can filter what you can and cannot do before making an API request.

Apify makes this process very simple on its server configuration page.

You can control which Apify actors your client will have access to, along with other parameters. In case of an attack, this prevents the hacker from being able to use your API to do too much harm.

Still, you need to provide the API key to your agent, along with the MCP URL, which in my case looks like this:

https://mcp.apify.com/?tools=docs,get-dataset,dadhalfdev/techcrunch-scraper-per-event,dadhalfdev/imovirtual-scraper-per-event

I’ve selected the TechCrunch scraper to extract tech articles. And the Imovirtual scraper to fetch property listings in Portugal.

For TechCrunch, I told the agent to do this:

Use this Apify actor:

dadhalfdev/techcrunch-scraper-per-event

Scrape up to 10 articles.

With the category “AI”

And summarize them to me here on Telegram.

The output:

1. Last 24h for +1 pass @50% off TechCrunch Disrupt 2026 (Jan 30)
Final chance to save $680 on Oct event w/ 250+ leaders (Barra, Khosla, Nadella).
Link (https://techcrunch.com/2026/01/30/last-24-hours-to-grab-your-1-pass-at-50-off-to-techcrunch-disrupt-2026/)
2. Tim Cook vague on Apple AI monetization (Jan 29)
Q4 revenue $143.8B up 16%; analyst presses on AI ROI—Cook: “creates great value.”
Link (https://techcrunch.com/2026/01/29/guys-i-dont-think-tim-cook-knows-how-to-monetize-ai/)
3. Musk: SpaceX/Tesla/xAI merger talks (Jan 29)
Early-stage; could IPO SpaceX or consolidate (xAI owns X). SpaceX valued $800B.
Link (https://techcrunch.com/2026/01/29/elon-musk-spacex-tesla-xai-merger-talks-ipo-reuters/)
4. Amazon eyes $50B OpenAI investment (Jan 29)
Amid $100B round valuing OpenAI @ $830B; despite Anthropic ties (AWS $8B invested).
Link (https://techcrunch.com/2026/01/29/amazon-is-reportedly-in-talks-to-invest-50-billion-in-openai/)

Make sure you use the correct inputs. For example, in the example above, I forgot to set the max_posts input, so the scraper kept running even after the first 10 articles were scraped and sent to my Telegram chat. If you don’t use all of the actor’s inputs, you may end up paying more than necessary.

This is a simple example of how you can access the latest news without opening social media or jumping between multiple websites. It provides instant access to information, anytime and anywhere.

Connect to Notion and manage tasks

For Notion, we are not going to use an MCP because it’s simpler with the API token, and it has an extra layer of security that Apify doesn’t. You can select the page where your client will have access, preventing a potential attacker from accessing other pages.

First, you need to create a new integration in Notion.

I named mine Dad Half Bot, and I set it up to only read content with no access to user information.

I sent the token to the agent:

I want you to connect to my Notion using this API:

ntn_346...

Then I moved to my Task List page and added the new integration (Dad Half Bot). Note that it won’t be able to access other pages without your permission.

I told it to access my ongoing tasks, and this is what I got:

This is a simple example, but the agent can be granted access to additional pages with different structure formats (databases, tasks, tables, text, etc). With write access enabled, tasks can be created and managed directly from Telegram, and you can stop using the Notion app on your phone if you want.

Reminders and notifications

Another straightforward use case is setting up reminders. This may seem obvious, but it’s a real need. How many of you miss email notifications? I do. But I never miss them on WhatsApp or Telegram when notifications are enabled, of course.

You can also create personalized reminders, and the bot will remember them indefinitely. Even though this is a simple implementation, there’s one important thing you need to know.

OpenClaw schedules prompts, so if you try this:

Say “I love you ❤️” daily at 7 am.

The bot will most likely reply:

“Oh, thanks, I love to ❤️!” daily at 7 am.

You need to do this instead:

Respond with ONLY this: “I love you ❤️” daily at 7 am.

The example above exists because I gave my girlfriend access to the bot, and now she receives a loving message every morning.

But the key idea is to think about how the bot should respond to your prompt, rather than what you would say yourself.

I’ve been using it for birthdays (better than Google Calendar), meetings, and for registering simple ideas that come to mind while I’m in the park with my daughter.

Need help with automation, AI, or data science? Let’s talk.

Book a 20-min consultation

Conclusion

In this piece, we’ve covered some of OpenClaw’s strongest features and explained why it can be a dangerous tool if not used correctly. It can be sensitive to prompt injection and may store APIs, tokens, and passwords in plain text files, which can be easily transferred to the wrong locations.

In short, the more tasks it handles, the less predictable it can become, and surprises may arise.

We’ve also covered the essential steps and commands to get started with OpenClaw without compromising safety, along with three straightforward use cases that can make everyday life easier.

This is a new tool, and the internet will soon be filled with more creative ideas. Hopefully, after reading this piece, you’ll be able to decide which of them are worth trying.

Some time ago, Google announced its plans to remove third-party cookies from Google Chrome, due to Google’s Privacy Sandbox initiative to minimize users’ exposure to surveillance advertising and reduce cross-site tracking.

The reactions to this rumor were, and still are, extremely polarised. On one hand, users who prioritize their privacy were happy with this decision. On the other hand, marketers saw it as a threat to the way they do business.

As of today, this measure hasn’t been fully implemented, with only a small percentage being affected. However, many browsers, such as Brave and Firefox, already block third-party cookies by default, and even Chrome is moving into a user-choice approach, making the overall cookie environment not a solid source of user information.

With user tracking becoming less effective, marketers cannot confidently determine which channels are driving ROAS (Return On Ad Spend).

This is where the Marketing Mix Modeling (MMM) plays a key role.

The MMMs analyse aggregated data (spending, impressions, reach, sales) across multiple channels, and they use time-series data to estimate how these activities influence the sales (organic and non-organic)

There are a few MMM frameworks worth exploring, such as Meta’s Robyn, PyMC Marketing, and Meridian, but in this article, we are going to focus on the latter, not only because it’s extremely user-friendly and open-source, but also because it clearly accounts for the combination between paid and organic channels.

We will start by understanding how Marketing Mix Modeling works, and then we’ll dive into the steps to train a model and use it to gather insights with Meridian’s built-in functions and dynamic charts.

How Marketing Mix Modeling works

Marketing Mix Modeling, as the name suggests, is an analytical approach that uses a “mix” of marketing inputs, such as spend, impressions, clicks, reach, promotions, and more, to estimate their impact on sales and revenue through Machine Learning (ML) algorithms.

The statistical models behind MMMs are typically multivariate regressions or Bayesian models. They rely on historical time-series data across all marketing channels, along with seasonality and trends.

Google’s Meridian framework follows a Bayesian approach, combining prior knowledge with observed data to estimate media effects while quantifying uncertainty.

The Bayesian regression model used by Meridian is based on Bayes` Theorem, which is represented by the following equation:

Where:

• P(A|B) (Posterior): The probability of A happening given that B is true.

• P(B|A) (Likelihood): The probability of seeing evidence B if A is true.

• P(A) (Prior): Your initial belief about A before seeing the evidence.

• P(B) (Evidence): The total probability of the evidence occurring under all possible scenarios.

So, how would you think of the Bayes Theorem applied to Marketing Mix Modeling?

Instead of a single observation, as the formula suggests, the theorem is used to estimate channel effects across time-series data. For instance, we may start with a prior belief about the expected contribution of a channel like Google Ads, informed by historical performance or domain knowledge.

The likelihood then incorporates observed data while accounting for factors such as seasonality, overlapping channels, and organic demand. By combining the prior with the likelihood, the model infers a posterior distribution over the channel’s contribution.

The theorem would look more like this:

The denominator (evidence) here represents the probability of seeing your actual sales data under every possible combination of channel effects, seasonality, and organic trends.

Why choose Meridian over other MMM frameworks

Now that we’re more familiar with Bayes’ Theorem, we can define our prior expectations about the types of questions Meridian is designed to answer. In practice, these typically include:

• How did each marketing channel contribute to revenue or another KPI?

• What was the return on ad spend (ROAS)?

• How should I allocate my marketing budget going forward to maximize impact?

• Where do diminishing returns begin for each marketing channel?

With Meridian, we will be able to answer all of these questions and more. These are some of the key features of this framework:

• Works with local, regional, and national data: Analyzes marketing performance at a regional or city level, not just nationally.

• Prior knowledge about channel performance: You can include what you already know about your marketing, such as results from past experiments.

• Accounts for diminishing returns and delayed effects: Saturation and lagged effects for paid and organic media are modeled using parametric transformation functions.

• Supports reach and frequency data: Beyond impressions, Meridian can utilize reach and frequency to gain a deeper understanding of how exposure impacts performance.

• Optimizes marketing budgets: You can determine the optimal budget allocation across channels based on your total.

• Runs “what-if” scenarios: You can simulate different marketing strategies, such as reallocating spend, and estimate how ROI would change under those scenarios.

• Evaluates model quality: You can assess how well the model fits historical data and how it performs on unseen data.

• Includes non-media and organic factors: You can optionally include factors such as pricing changes or promotions.

In the next chapter, we are going to see how to train a model using a dataframe with both organic and paid channels.

How to train the Meridian MMM

Before training the data, the first question that arises is to decide which time granularity makes more sense to use: daily, weekly, or monthly.

Training an MMM with daily data would not only take a huge amount of time, especially considering several years, but could also lead to overfitting, and a monthly or quarterly granularity would remove a lot of information regarding seasonality and trends, leading to underfitting.

Therefore, a weekly granularity would be the best approach, but it also depends on the number of features in your dataset and the number of data points you have. According to several articles, there’s a rule of thumb in Marketing Mix Modeling: you should have 10 data points (rows) per feature. So, using weekly granularity would be recommended if you’re using, for instance, 2 years of data (around 104 weeks) and 10 features in your dataset.

In this piece, we are going to work with 3 years of information, so I grouped the data at a weekly level, but I’ve also added information about the month and the quarter because it helps to separate true marketing effects from seasonality and trends.

We are now going to train the dataset above, which includes media spend, non-spend, and organic channels.

Install and set up Meridian

The easiest and fastest way to train the data is by using a Google Colab notebook with a remote GPU. For that, you need to open a notebook and in the top corner, change the runtime type to T4 GPU.

Now, in the first cell of your notebook, you can connect to Google Drive to directly store and fetch data from there.

from google.colab import drive
drive.mount(’/content/drive’)

To install Meridian, you can run this pip command:

!pip install --upgrade google-meridian[colab,and-cuda]

Once installed, we can start by structuring the features of the dataset like this:

MEDIA = [
    “instagram_impressions”,
    “x_impressions”,
    “google_clicks”,
    “tik_tok_impressions”,
    “facebook_impressions”]

MEDIA_SPEND = [
    ‘instagram_spend’,
    ‘x_spend’,
    ‘google_spend’,
    ‘tik_tok_spend’,
    ‘facebook_spend’]
ORGANIC_COLS = [
    ‘direct_sessions’,
    ‘organic_impressions’,
    ]
CONTROL_COLS = [’week’, ‘month’, ‘quarter’,  ‘year’]
mapping_media_spend = {
    ‘instagram_spend’: ‘Instagram’,
    ‘x_spend’: ‘X’,
    ‘google_spend’: ‘Google’,
    ‘tik_tok_spend’: ‘Tik Tok’,
    ‘facebook_spend’: ‘Facebook’
}
mapping_media = {
    “instagram_impressions”: “Instagram”,
    “x_impressions”: “X”,
    “google_clicks”: “Google”,
    “tik_tok_impressions”: “Tik Tok”,
    “facebook_impressions”: “Facebook”
}

The MEDIA list contains the user activity, like impressions and clicks. The MEDIA_SPEND list contains the same channels but focuses on spending. Both MEDIA_SPEND and MEDIA are linked through mapping_media_spend and mapping_media, so make sure to have column values for both dictionaries. The ORGANIC_COLS list contains the features that do not require spending. The CONTROL_COLS are all columns related to time.

We can use the above lists and dictionaries to create a mapping object for the model:

from meridian.data import load

coord_to_columns = load.CoordToColumns(
    time=’week’,
    controls=CONTROL_COLS,
    kpi=’revenue’,
    media=MEDIA,
    media_spend=MEDIA_SPEND,
    organic_media=ORGANIC_COLS
)

The KPI is what you’re trying to predict (the target), which in our case is the revenue.

Create the priors

Now comes the trickiest part of the Marketing Mix Modeling steps, which is creating the priors. As we have seen before, the priors, just like the posteriori, are distributions, and they represent the information you provide to the model before it “sees” the data.

A prior can come from multiple sources, including results from past experiments, industry benchmarks, previous MMM results, and your team’s subject matter expertise. In this tutorial, we don’t have previous information about the data, so we are going to use the default priors provided by Meridian.

You can read more about the priors (beliefs) here.

To calculate the default priors, we are going to use the PriorDistribution object from the Meridian library. In addition, we will use the tensorflow_probability package to create distributions.

pip install tensorflow-probability

We won’t do any modifications on the default input values, but I’m still going to display them and add comments so you can understand them better:

import tensorflow_probability as tfp
from meridian.model import spec, prior_distribution

prior = prior_distribution.PriorDistribution(
    # LogNormal is used because ROI must be positive.
    # These are Meridian’s recommended defaults when no prior knowledge exists.
    roi_m=tfp.distributions.LogNormal(
        loc=0.2,    # log-mean
        scale=0.9   # log-std (fairly wide -> weakly informative)
    ),
    roi_rf=tfp.distributions.LogNormal(
        loc=0.2,
        scale=0.9
    ),
    # Organic media uses contribution priors.
    # Beta(1, 99) implies a small expected contribution per channel,
    contribution_om=tfp.distributions.Beta(
        concentration1=1.0,
        concentration0=99.0
    ),
    contribution_orf=tfp.distributions.Beta(
        concentration1=1.0,
        concentration0=99.0
    ),
)

The default values are the most flexible when we don’t have any clue about the data, but they can be adjusted accordingly. The model specification will look like this:

from meridian.model import spec

model_spec = spec.ModelSpec(
    # Paid media is modeled using ROI priors
    media_prior_type=”roi”,

    # Organic media is modeled using contribution priors
    organic_media_prior_type=”contribution”,

    # Attach the prior configuration
    prior=prior,
)

Load and train

Now we can create the loader object that takes the coordinates and the mappings previously created for all paid and organic channels:

from meridian.data import load

loader = load.DataFrameDataLoader(
    df=df,
    kpi_type=’revenue’,
    coord_to_columns=coord_to_columns,
    media_to_channel=mapping_media ,
    media_spend_to_channel=mapping_media_spend
)
data = loader.load()

With the loader and the model_spec, we’re ready to instantiate the model:

from meridian.model import model

mmm = model.Meridian(input_data=data, model_spec=model_spec)

Since Meridian is based on a Bayesian model, to train it, we need to sample from the posterior distribution using Markov Chain Monte Carlo (MCMC).

# Initialize and run the model
mmm.sample_prior(500)

# Sample the posterior)
# This is the step that “learns” from your data
mmm.sample_posterior(
    n_chains=4,    # Number of parallel sampling chains (recommend 4-7)
    n_adapt=500,   # Adaptation steps to tune the sampler
    n_burnin=500,  # “Warm-up” samples to throw away
    n_keep=1000,   # Actual samples to keep for your results
    seed=101       # For reproducibility
)

After running the above cell (may take a bit), make sure you save your mode as a pkl file, so you don’t need to run it again:

model.save_mmm(mmm, “/content/drive/MyDrive/mmm_model.pkl”)

Get insights from the Meridian model

That’s, you’ve waited a few minutes, and now it’s time to see some results!

The first thing we can do, if you saved the model, is to load in another empty notebook:

mmm = model.load_mmm(”/content/drive/MyDrive/mmm_model.pkl”)

Let’s see how well our model fits the data:

from meridian.analysis import visualizer

model_diagnostics = visualizer.ModelDiagnostics(mmm)
model_diagnostics.predictive_accuracy_table()

In our case, the R-squared is 0.8. The higher this value, the better the model fitting. However, values that are too high can also indicate overfitting.

When it comes to charts, there are several options. The first chart we can look at is the total spend versus the contribution:

media_summary = visualizer.MediaSummary(mmm)
media_summary.plot_spend_vs_contribution()

The above chart only takes into account the paid marketing channels. Among them, Instagram shows the highest ROI or ROAS, but it was also the channel with the highest spend. On the other hand, X didn’t require as much money invested to also generate a high ROI. The remaining channels show relatively positive ROI, suggesting potential to scale.

Now, to get the contribution, let’s use a waterfall chart.

media_summary.plot_contribution_waterfall_chart()

This chart illustrates how total revenue is distributed across both organic and paid marketing channels. In this chart, the baseline is driving the most, meaning that 58.2% of the revenue is not driven by the channels, but by seasonality and other sources. Instagram is the dominant media contributor, accounting for 26.7% of total revenue, followed by X with 10.3%. The organic channels have very little impact on this data sample.

Need help with automation, AI, or data science? Let’s talk.

Book a 20-min consultation

Conclusion

We could continue generating additional charts, such as adstock decay or diminishing returns. Meridian offers multiple visualizations, but the main goal here was to demonstrate how to create and fine-tune the model, while showcasing a few examples of what an MMM can deliver.

While Meridian includes several built-in charts, you can also extract the dataframes that power these plots and use them to build your own custom charts and dashboards.

Meridian is a very powerful open-source model that, as you’ve seen, significantly simplifies the Marketing Mix Modeling process, without requiring deep knowledge of complex mathematics or statistical functions.

If you are a marketer or data scientist who wants to avoid dependence on cookies, using Meridian, or any other MMM framework, is the way to go.

This growing popularity has even pushed major players like Meta and OpenAI to launch dedicated apps for creating and sharing AI-generated videos, such as Sora and Meta AI.

Most users currently consume AI videos primarily for entertainment rather than for news or reliable information. In fact, the same report shows that users are generally less likely to trust content on social media when it is AI-generated.

Nonetheless, the trust in AI content may shift with the introduction of avatars, regardless of whether they are realistic or not. If the creator can validate that the information shared by the avatar is indeed correct, people will follow and support the page, even if it is not a real person talking.

In this article, I’ll guide you through two methods for creating avatar videos using Kling AI. The first method uses Kling’s Motion Control, which replicates the movements of a real person and applies them to a character or image you provide. The second method combines text, an image, a prompt, and audio to generate an avatar video.

We’ll start by generating videos directly from the platform’s console, then move on to building a workflow that automatically generates and publishes them to social media using the Kling AI API and Python.

Create your first avatar video

You can create a free account on Kling AI, but video generation on the platform comes with a cost. New users receive a limited number of credits after registering, which are usually only enough to generate images or very short video clips.

Kling offers two main pricing options: a monthly subscription or the ability to purchase credits separately. API access is priced differently and will be discussed in a later section.

I bought a few credits some months ago, but if you’re only starting, the Standard plan has more perks than simply acquiring credits. That’s because buying credits won’t remove the watermark in your videos, and $5 only gets you 330 credits, while with the Standard plan, you will get up to 660 per month.

⚠️ NOTE: If you’re planning on using the API instead, skip the above pricing options. Otherwise, you’ll be paying for using the platform and also for the API.

Before picking a pricing plan, you can see if you have enough credits to make your first experiment with the Motion Control model under the AI Video Generator tab.

This option lets you upload a reference video along with an image, which Kling AI uses to replicate the video’s movements and expressions. The number of credits required depends on the length of the reference video. Make sure you start with a short clip.

In addition, you must provide a prompt that supplies contextual and behavioral guidance for the output. You can make your avatar more extroverted and smiley or more serious, depending on how you structured your prompt.

Here’s an example with an image I created using Nano Banana Pro on Gemini:

The Motion Control requires you to record yourself or have a video reference. The other approach is to create an avatar that, instead of taking a reference video, takes a text, an audio, an image, and a prompt. On Kling AI, you can do this by selecting Video and then Avatar.

On the console, you’ll find several avatar examples ready to be used, but that’s not as fun as creating your own!

The steps are similar to Motion Control, but you need to add the speech (text) and an audio. The longer the speech, the more credits will be consumed. Instead of the text, you can also upload an audio file.

In the next section, we will use the Avatar 2.0 to automatically create informative videos based on summaries from articles. For that, we will need the Kling AI API and the Python programming language.

Create an authentication token for the Kling AI API

You can use the same Kling account to access the API console.

The pricing plans for the API usage are different than the ones previously discussed. In our case, we’re going to use the pricing that is visible under the Functional Models, which includes the Avatar, Lyp-Sync, and other products.

These are the available pricing packages:

Once you’ve picked your preferred package, you can create an API key, which has an Access Key and a Secret Key.

Now, you still won’t be able to make an API request with the two, you must do a JWT verification. Here’s the Python script:

import time
import jwt

access_key = “AP4...” # fill access key
secret_key = “adJme...” # fill secret key

def encode_jwt_token(ak, sk):
    headers = {
        “alg”: “HS256”,
        “typ”: “JWT”
    }
    payload = {
        “iss”: access_key,
        “exp”: int(time.time()) + 1800, # The valid time, in this example, represents the current time+1800s(30min)
        “nbf”: int(time.time()) - 5 # The time when it starts to take effect, in this example, represents the current time minus 5s
    }
    token = jwt.encode(payload, secret_key, headers=headers)
    return token

authorization = encode_jwt_token(access_key, secret_key)
print(authorization) # Printing the generated the authentication token

You can tweak the script to increase the expiration time. The output is a Bearer token that will be used for the API requests.

You can learn more about this and the following steps by reading the official API documentation here.

Create the audio and the avatar with the Kling AI API

Unlike the platform interface, where an avatar video can be created in just a few steps, using the API requires a multi-step workflow. You must first generate an authentication token (as explained earlier), then create the audio file, generate the video, and finally retrieve the video URL.

Let’s start by creating the audio file. In this case, we will choose a Voice ID from the spreadsheet provided by Kling AI.

import requests

AUDIO_URL = “https://api-singapore.klingai.com/v1/audio/tts”
API_TOKEN = “eyJhbGciOiJIUzI1Ni...”


headers = {
    “Authorization”: f”Bearer {API_TOKEN}”,
    “Content-Type”: “application/json”
}
payload = {
    “text”: (
        “Forget small steps—Ethereum is preparing for a massive leap. “
        “By 2026, the network is set to start scaling exponentially “
        “thanks to Zero-Knowledge (ZK) technology. Think of it as a total “
        “brain transplant for the blockchain, “
        “similar in scale to ‘The Merge’.”
        “Instead of every validator re-doing every single transaction, “
        “they’ll soon just verify tiny ZK-proofs. “
        “This shift could eventually skyrocket Ethereum from 30 “
        “transactions per second to a staggering 10,000. “
        “The best part? It’s so efficient you could theoretically “
        “run a node on a smartphone. While it’s a multi-year journey, “
        “2026 is when the real speed revolution begins.”),
    “voice_id”: “chat_0407_5-1”,
    “voice_language”: “en”,
    “voice_speed”: 1.2,

}

response = requests.post(AUDIO_URL, headers=headers, json=payload)
print(response.json())

The code snippet above takes a text from a Cointelegraph article as input, along with the voice_id, the voice_language and the voice_speed. The output is the URL for the voice file and the audio id. The response looks like this:

{’code’: 0, ‘message’: ‘SUCCEED’, ‘request_id’: ‘f14a385d-20b8-45a8-8469-b00d2aaded65’, ‘data’: {’task_id’: ‘835047569842642979’, ‘task_status’: ‘succeed’, ‘task_result’: {’audios’: [{’id’: ‘835047569859411990’, ‘url’: ‘https://v16-kling-fdl.klingai.com/bs2/klingai-kling-input-stunt-sgp/muse/834731746867421228/AUDIO/20251231/d147f6fa9948f358602fa42cd956bb21-f9dc5b57-6d5a-4e0b-8f37-fe06572484b0?cacheKey=ChtzZWN1cml0eS5rbGluZy5tZXRhX2VuY3J5cHQSsAFGHCN_CgjsePaPU4pF4kLs3wTWpJBrE6KgbqzxUmj753zYHmfUr372mu1-CGXxj0PJ0GTW-XLYGUjUYGHx2uENIrR53Ud_K8H3JnHMj-1YFkqx8dEVNnvpTVqylkCUqCEujdgRNETYMqctNaJdWzg3ymVJGWPQ-Ur5QKyoFPpmWzP9If7YAeN3pRXlpTYjIZNM68E_kZeGIzUAMBv2Ml60ebpaIkk9yuBeMvh8DLZBbxoSOFvh80_9xnI_XufDmu_AA0PdIiA1c1FjVWNODooKSYUEUJ6KO8OkJ3eiwBnzMN53F6ne9igFMAE&x-kcdn-pid=112781&ksSecret=7ba1734ea6d1d4e041b846623705b975&ksTime=697ba710’, ‘duration’: ‘39.384’}]}}}

Now we’re going to use the audio id and an image to generate the video.

This is the avatar image I used in my video:

Here’s the script to generate the video:

import requests
import base64

with open(”character_female.png”, “rb”) as image_file:
    image_data = image_file.read()
    img_base64 = base64.b64encode(image_data).decode(”utf-8”)

headers = {
    “Authorization”: f”Bearer {API_TOKEN}”,
    “Content-Type”: “application/json”
}

payload = {
    “image”: img_base64,
    “audio_id”: 835047569859411990,
    “mode”: “std”,
    “prompt”: (
        “Create a realistic talking avatar with a natural smile and focused”
        “, confident expression, speaking directly to the camera.”
        “\n\nCamera & Background: Keep the camera fixed and do not move or “
        “alter the background in any way.\n\n”“Body Movement:\n”
        “Add subtle, natural upper-body movements (slight torso “
        “shifts, gentle head tilts, minimal shoulder motion) to avoid a “
        “static or robotic appearance.\n\n”
        “Performance style: Calm, engaging, and professional—natural speech “
        “rhythm, smooth facial animations, and realistic eye and “
        “mouth movements.\n\n”
        “Overall quality: Photorealistic lighting, stable framing, “
        “and lifelike motion without exaggeration.”)
}

response = requests.post(
    “https://api-singapore.klingai.com/v1/videos/avatar/image2video”,
    headers=headers,
    json=payload)


print(response.json())

After running this script, the video may take some time to become available, depending on the audio length. Be sure to add a delay before proceeding to the fetch step. This snippet produces the following output:

{’code’: 0, ‘message’: ‘SUCCEED’, ‘request_id’: ‘d6b40d68-6866-4b31-849e-a301036d0b14’, ‘data’: {’task_id’: ‘835047984131473475’, ‘task_status’: ‘submitted’, ‘created_at’: 1767119472638, ‘updated_at’: 1767119472638}}

In the next and final step, we’re going to use the request_id and task_id to obtain the video URL. Here’s how:

request_id = “d6b40d68-6866-4b31-849e-a301036d0b14”
task_id = ‘835047984131473475’

url = f”https://api-singapore.klingai.com/v1/videos/avatar/image2video/{request_id}”


headers = {
    “Authorization”: f”Bearer {API_TOKEN}”,
    “Content-Type”: “application/json”
}

payload = {
    “task_id”: task_id,
}

response = requests.get(
    url,
    headers=headers,
    json=payload)


print(response.json())

The response has the video URL from where you can download the video:

{’code’: 0, ‘message’: ‘SUCCEED’, ‘request_id’: ‘151d0403-a017-42af-89af-668d0c2ff53a’, ‘data’: {’task_id’: ‘835047984131473475’, ‘task_status’: ‘succeed’, ‘task_info’: {}, ‘task_result’: {’videos’: [{’id’: ‘835047984186015791’, ‘url’: ‘https://v16-kling-fdl.klingai.com/bs2/upload-ylab-stunt-sgp/muse/834731746867421228/VIDEO/20251231/7bc028554e69b8b6c9527496bdfc00e6-901b7d5a-40c3-4f54-b4da-d499cfe599d7.mp4?cacheKey=ChtzZWN1cml0eS5rbGluZy5tZXRhX2VuY3J5cHQSsAH1weioZR4lXm4IjZ4hSNdgLYXSg-qYje0OYPNG_RCV-L7ISZHkFYzkwOvZ-Q2YpAt514WqvbwsB7Y648-4JFYOg0pQV_9rn2AL6UgS2MPLQDcnjSwhk4URxBFL9kKceOwExCp7a2LwoJ7MSjYgJn1HOi661_f17mkhyCzCGmGQcjsrM7x8-7IUb6iLMJ95L8jrGaln6YfxLxwRLHwHOzK2MrP334DYiQaJW7p_KrmcDhoSv3Uc89R3_xETrF65D8P8-8MOIiD_d69m8PnwyzUGrpf3N1pm6gCX1XJxFgrwpIp9kHIlCigFMAE&x-kcdn-pid=112781&ksSecret=29b1f4a65600c88cac188b609446c9bc&ksTime=697baad9’, ‘duration’: ‘39.466’}]}, ‘task_status_msg’: ‘’, ‘created_at’: 1767119472638, ‘updated_at’: 1767120346646}}

Here’s the final output:

This video alone consumed around 16% of the total points available (I’m using the cheapest API pricing plan). This means that with $9.79, I can generate up to six videos, if they all have the same length.

Automate posts on social media

If you want to automate social media posts on platforms like X, Bluesky, and Instagram, you can use either cron jobs on your local machine or a remote VPS.

I prefer running automation and web scraping on a Contabo VPS, using my local machine only for testing. This keeps my main workstation free and avoids unnecessary RAM and resource usage.

I’m currently using the Cloud VPS 30 for only €11.2/month, which has 24 GB RAM (more than my laptop), 8 vCPU Cores, and 400 GB SSD. This plan is more than enough for most automation tasks, but feel free to have a look at the other VPS and VDS options available.

When it comes to social media posting, I’ve previously written two articles. One for X (former Twitter) and another for BlueSky.

Create a bot to publish on X

Create a bot to publish on BlueSky

If you want to build an avatar like mine that talks about crypto, tech, and world news, you’ll need a web scraper to extract fresh daily content. This content can then be summarized and used as the audio script for your avatar.

If you’re not familiar with summarizing content using LLMs and Python, take a look at the article above on building bots for X.

As for the web scraper, if you’re short on ideas, you can use one of mine:

• Aljazeera Scraper: Extract articles from Aljazeera, providing essential insights into current events and regional tensions.

• TechCrunch Scraper: Get the latest tech and AI news from TechCrunch.

• Cointelegraph Scraper: Extract data from Cointelegraph, a leading platform for cryptocurrency and blockchain-related news.

You can find many other scrapers available on Apify.

Need help with automation, AI, or data science? Let’s talk.

Book a 20-min consultation

Conclusion

The costs associated with creating a single avatar video can quickly add up. For example, with less than $10, I would only be able to publish six videos in total.

That said, this expense is relatively small when compared to the time and effort required to produce a traditional social media video. Recording myself talking about the latest crypto news would involve multiple takes, video editing, and careful audio adjustments.

All of these manual steps are far more costly than a couple of dollars per video.

An avatar that consistently generates content can instead be used to drive traffic to your website, support affiliate marketing, or power other growth strategies.

As we’ve seen at the beginning of this article, users are consuming more AI-generated videos than ever. The real challenge is creating an engaging, value-driven avatar. One that can generate passive income while you sleep.

csfwgwg

A key element in this is Artificial Intelligence (AI), which requires vast amounts of data for model training and therefore relies on massive crawlers running nonstop. At the same time, AI has also made web scraping more accessible, allowing non-developers to collect data through vibe coding and seamless APIs.

According to a report by Market Research Future, the web scraper software market was valued $1.01 billion in 2024. In 2025, the Mordor Intelligence report shows that the same market is now valued at $1.03 billion, a steady 1.98% increase in just one year. The same report projects that the web scraping market will reach $2 billion by 2030, nearly 98% growth in just six years.

While web scraping is more accessible with the integration of AI, many websites have also shielded themselves against massive bot requests. Making the job difficult for many APIs and web scrapers powered by vibe coding scripts.

In this piece, I’m sharing with you 5 tips that Large Language Models (LLMs) don’t tell you to build a robust web scraper that can outperform 90% of the websites with only one tool: Playwright.

First steps with Playwright

At the time I’m writing, I have over 20 Apify actors running, all built using Playwright.

Before, I used to web scrape with Selenium, Beautiful Soup, and Scrapy, but I’ve realized that I can have successful bots only using Playwright, so why complicate things?

Playwright is a headless browser considered to be faster than its competitors and with better support for modern websites. Documentation is easy to follow and supports multiple languages.

To install Playwright, you need the following pip command:

pip install playwright

To install the supported browsers, use this command in your terminal:

playwright install

Now, the basic component of Playwright is often called page and it is instantiated this way:

from playwright.async_api import async_playwright


async with async_playwright() as pw:

    # creates an instance of the Chromium browser and launches it
    browser = await pw.chromium.launch(headless=False)

    # creates a new browser page (tab) within the browser instance
    page = await browser.new_page()

I will not cover all of Playwright’s functions in this piece, but I will highlight the ones I use in most of my crawlers and have proven to be indispensable:

# Go to page and wait to render full content: images, APIs and more
await page.goto(”some_url”, wait_until=”networkidle”)

# Go to page and wait only for DOM structure (initial HTML). Much faster.
await page.goto(”some_url”, wait_until=”domcontentloaded”)

# Create some timing between actions
await page.wait_for_timeout(3000)

# Get text content only
await page.locator(”a”).text_content()

# Get an attribute
await page.locator(”a”).get_attribute(”href”)

# Get all elements with the same class
elements = await page.locator(”a.font-big”).all()

# Click the first element
await page.locator(”a.font-big”).first.click()

# Click the last element
await page.locator(”a.font-big”).last.click()

# Click element by index
await page.locator(”a.font-big”).nth(3).click()

In addition to these examples, it is also useful to use page.get_by_role(), page.get_by_title(), page.get_by_label(), and so on, whenever you can. This is usually more straightforward than using the page.locator().

For more about Playwright’s functions, visit the well-structured official documentation.

Now we will have a look at the 5 tips to improve your web scraping skills while integrating Playwright. They aim at answering the following questions:

1. How to increase the stealth performance?

2. How to intercept the website’s HTTP response?

3. How to handle infinite scroll websites?

4. How to target HTML elements accurately?

5. How to make proper proxy rotation?

Let’s get started!

1 - How to increase the stealth performance?

If you’re familiar with Playwright, you’ve probably heard of playwright_stealth.

While it can work for some websites, it is definitely an outdated solution to improve your stealth performance.

The alternative: Camoufox.

This open-source anti-detect browser can be integrated with the Playwright API. It works by providing Camoufox with detailed device characteristics, including the operating system, device information, navigator properties, headers, screen size, and more.

To install Camoufox, use this pip command:

pip install -U camoufox

Your playwright’s page object can be instantiated in the following way:

from camoufox import AsyncCamoufox

async with AsyncCamoufox(
    os=”macos”,
    humanize=True,  # Enable humanized cursor movement
    headless=False,  # Keep visible for debugging
    window=(1280, 720),  # Set window size
) as browser:

  page = await browser.new_page()

Using this browser won’t solve all your issues. You may still get blocked by websites protected by Datadome, but it is significantly better than using Playwright only.

You can also add proxies, tweak the parameters, and add BrowserForge to see if the stealth is improved.

2 - How to intercept the website’s HTTP response?

Many developers who engage in web scraping tend to strictly extract and parse the HTML content. While this solution works, it may not be the fastest way to crawl data, and you may miss important metadata and features.

Many modern websites don’t load all their data in the initial HTML. Instead, they fetch data dynamically from APIs (such as JSON and GraphQL) after the page loads.

Therefore, before tackling the HTML content, check if the website has an API response. This can be triggered by simply loading the page, clicking on buttons, or scrolling the page.

Let’s take the Hackernoon website as an example.

In this case, there’s a JSON API response for each article that contains a lot of useful information about it, including the title, the slug, the ID, the author’s name, the image, and even the article’s body.

To extract the information from the JSON, you need to place this code before the event is triggered:

# Create set to track processed article IDs
processed_ids = set()

# Get response and set up response listener
response_received = asyncio.Event()
page.on(
    “response”,
    lambda response: check_json(
        response, response_received, processed_ids))

Now you can create a function to save the content in JSON format or CSV:

async def check_json(response, response_received, processed_ids):
    “”“ Get data from HackerNoon JSON response “”“

    url_response = (”https://hackernoon.com/_next/data/”)

    if url_response in response.url:
        try:
            items = await response.json()
            article_data = items[’pageProps’][’data’]
        except Exception as e:
            return

        if article_data:
            response_received.set()

            # Extract article features based on example.json structure
            try:
                article_id = article_data.get(’id’)
                
                # Check for duplicates using article ID
                if article_id in processed_ids:
                    return
                
                # Add to processed IDs set
                processed_ids.add(article_id)
                
                profile = article_data.get(’profile’, {})
                
                # Extract stats data
                stats = article_data.get(’stats’, {})
                
                post_data = {
                    ‘id’: article_id,
                    ‘title’: article_data.get(’title’),
                    ‘slug’: article_data.get(’slug’),
                    ‘link’: “https://hackernoon.com/” + \
                        article_data.get(’slug’),
                    ‘excerpt’: article_data.get(’excerpt’),
                    ‘tldr’: article_data.get(’tldr’),
                    ‘articleBody’: article_data.get(’articleBody’),
                    ‘createdAt’: article_data.get(’createdAt’),
                    ‘mainImage’: article_data.get(’mainImage’),
                    ‘mainImageHeight’: article_data.get(’mainImageHeight’),
                    ‘mainImageWidth’: article_data.get(’mainImageWidth’),
                    ‘socialPreviewImage’: article_data.get(
                        ‘socialPreviewImage’),
                    ‘parentCategory’: article_data.get(’parentCategory’),
                    ‘tags’: article_data.get(’tags’, []),
                    ‘commentsCount’: article_data.get(’commentsCount’),
                    ‘pageViews’: stats.get(’pageviews’),
                    ‘arweave’: article_data.get(’arweave’),
                    ‘author_name’: profile.get(’displayName’),
                    ‘author_handle’: profile.get(’handle’),
                    ‘author_avatar’: profile.get(’avatar’),
                    ‘author_bio’: profile.get(’bio’),
                    ‘author_isBrand’: profile.get(’isBrand’),
                    ‘author_isTrusted’: profile.get(’isTrusted’)
                }

                # Save “post_data “data to JSON or CSV
                    
            except Exception as e:
                print(”Error extracting article data: %s”, str(e))   

Where do you see the URL response? You can find it in Headers under Network by inspecting the website’s page.

The Response tab under Network displays the JSON structure with all the fields you can scrape.

Now you know, next time, check this before engaging in intense HTML parsing, and note that the APIs can be triggered in different ways. Make sure to try buttons, filters, pagination, and scrolling.

This is how I built my Hackernoon Apify actor. You can try it yourself here.

3 - How to handle infinite scroll websites?

Many modern websites use infinite scroll to improve loading speed and improve the user experience. But they are definitely not loved by web scrapers.

Dealing with infinite scrolling can be challenging because websites tend to use different strategies to load their content. Scrolls can be located on different sides of the page. Some pages might not load content if you don’t move to the exact place on the page, others might require clicking to load more content, and so on.

The simplest way to scroll a page with Playwright is by using the wheel function:

n_scrolls = 10
for i in range(n_scrolls):
      await page.mouse.wheel(0, 15000)
      print(’scrolling’)
      await page.wait_for_timeout(3000)
      await page.mouse.wheel(0, -1500)
      print(’move up’)
      await page.wait_for_timeout(3000)

The reason why I’m using a loop in the example above is that most pages with infinite scroll take some time to load the hidden content. Therefore, it may not work if you just scroll to the bottom of the page. With the loop, you can ensure the new data is loaded in each iteration.

While this strategy works, sometimes you can’t trigger the loading this way. The best approach is to move the cursor to the nearest element that triggers it. In the case of a news website, most likely the last visible element is an article/card, so that’s a good reference to scroll down to.

Let’s see how to implement it.

# scroll to the item
location = page.locator(”article”).last
await location.scroll_into_view_if_needed()

With the two code snippets above, you should be able to handle most of the infinite scroll websites. Note that the more you scroll, the harder it gets to load, and your scraper can get very slow.

4 - How to target HTML elements accurately?

Sometimes using the page.locator() can be a real frustration. You try several classes, attributes, and text, and still the script prompts an error.

For those cases, my first approach is to target the main classes. Meaning that if you have an element with a lot of classes, start by tackling it with only one or two classes. This will make Playwright prompt an error similar to this:

      playwright._impl._errors.Error: Locator.text_content: Error: strict mode violation: locator(”div.container-Qnseki”).first.locator(”a.pencraft.pc-reset”) resolved to 2 elements:
          1) <a data-testid=”post-preview-title” href=”https://newsletter.theaiedge.io/p/adalflow-a-pytorch-like-framework” class=”pencraft pc-reset color-pub-primary-text-NyXPlw font-pub-headings-FE5byy clamp-y7pNm8 clamp-3-lxFDfR reset-IxiVJZ”>AdalFlow: A PyTorch-Like Framework to Auto-Optimi…</a> aka get_by_role(”link”, name=”AdalFlow: A PyTorch-Like”)
          2) <a href=”https://newsletter.theaiedge.io/p/adalflow-a-pytorch-like-framework” class=”pencraft pc-reset color-primary-zABazT line-height-20-t4M0El font-text-qe4AeH size-15-Psle70 clamp-y7pNm8 clamp-2-kM02pu reset-IxiVJZ”>AI Agent frameworks are becoming just as importan…</a> aka get_by_role(”link”, name=”AI Agent frameworks are”)

Since it doesn’t know which element you are referring to, it will kindly present you with multiple options.

From those options, you can use the recommended Playwright function to grab it, or use nth() with the right index.

If your selector still doesn’t work, switching to XPath inside the page.locator() function can often deliver more accurate results.

This is because XPath allows you to navigate the structure of the DOM directly.

Let’s take a look at this example.

Suppose you’re trying to target a button with a dynamic class name that changes on every page load. A CSS selector like this might fail:

button = page.locator(”.dynamic-button-class”)

Instead, you can use an XPath expression that targets the element by its text content or position:

button = page.locator(”//button[contains(text(), ‘Submit’)]”)

There are also Playwright built-in functions that mimic XPath:

button = page.locator(”button”).filter(has_text=”Submit”)

You can either decide to learn the XPath expression language or explore more Playwright’s documentation. But avoid very long XPath expressions to optimize code readability.

5 - How to make proper proxy rotation?

Some of you are familiar with proxies and proxy rotation, but the real challenge arises when choosing the right proxy for your needs.

If anonymity and avoiding raising flags are your top priorities, residential proxies are the best option. However, if you are web scraping a website that doesn’t have strong blocking measures, you may want to prioritize speed, and datacenter proxies are a better solution.

I rely on proxy rotation when scraping more challenging websites, and IPRoyal’s trustworthy residential proxies have consistently met my needs.

IPRoyal proxies are also one of the cheapest in the market, with an excellent score on review platforms such as G2 and Trustpilot. You can create an account in a few seconds, without requiring too much personal information, and you can even pay with cryptocurrencies, if you don’t want to have your credit card attached to it.

Once logged in, you have access to a user-friendly dashboard where you can select the best proxy option for your needs. If you don’t know what works best, you can answer a short survey to help you out.

If you decide to use residential proxies like me, you’ll see a page with your username, password, hostname, port, and several filters that allow you to customize your proxy’s location.

To implement proxy rotation, I select up to 7 different countries and cities, and I add them to a Python list in my script. It looks like this with Camoufox:

proxy_list = [
    None,  # No proxy first
    “<key>_country-fr_city-toulouse”,
    “<key>_country-es_city-valencia”,
    “<key>_country-pt_city-lisbon”,
    “<key>_country-es_city-madrid”,
    “<key>_country-es_city-huelva”,
    “<key>_country-es_city-sevilla”,
]

for proxy_element in proxy_list:
    # Configure browser settings based on whether using proxy or not
    browser_config = {
        “os”: “windows”,
        “humanize”: False,  # Enable humanized cursor movement
        “headless”: False,  # Keep visible for debugging
        “window”: (1280, 720),  # Set window size
    }
    
    # Add proxy configuration if proxy_element is not None
    if proxy_element is not None:
        browser_config.update({
            “geoip”: True,
            “proxy”: {
                “server”: “geo.iproyal.com:11201”,
                “username”: “<my_username>”,
                “password”: proxy_element
            },
        })
    
    async with AsyncCamoufox(**browser_config) as browser:
      # If page is accessed continue the script, if not try another proxy 

      

While proxy rotation is beneficial to bypass anti-bot measures, it slows down your web scraper’s speed. Therefore, if speed is a top requirement, you can rent a Virtual Private Server (VPS), so you can run your script with multiple instances and at the same time, one for each proxy.

I use Contabo’s Cloud VPS 30 for only €12.5/month, which has 24 GB RAM (more than my laptop), 8 vCPU Cores, and 400 GB SSD. This plan is more than enough for most automation tasks, but feel free to have a look at the other VPS options available.

With my Contabo VPS, I can run concurrent web scrapers and schedule them using cron jobs at different times. This approach helps prevent my IP from being flagged while also conserving RAM for my primary tasks.

Need help with automation, AI, or data science? Let’s talk.

Book a 20-min consultation

Conclusion

In this piece, we’ve covered the importance of Playwright headless browser for web scraping, and how it serves most of the web scraping tasks. Showing that you don’t need to learn multiple web scraping frameworks to build successful and performant crawlers.

Then we highlighted 5 different tips that vibe coding tools don’t tell you, but they make all the difference to turn you from a beginner to an advanced web scraping developer.

First, we discussed how to increase stealth and avoid being blocked by integrating Camoufox with Playwright.

Next, we looked at how intercepting HTTP responses before parsing HTML can save valuable time and help you capture hidden or critical data that might otherwise be missed.

We then examined how to handle infinite scroll pages efficiently using Playwright’s built-in functions, followed by strategies for optimizing data extraction with XPath or Playwright’s native selector capabilities.

Finally, we explored the differences between datacenter and residential proxies, highlighted the best options to start with, and explained how to integrate them effortlessly with Camoufox.

As demand for web scraping continues to grow, you can start applying these tips to land freelance projects on platforms like Upwork, build and monetize Apify actors for passive income, or simply create efficient crawlers for your own projects.

Perhaps this article will contribute to increasing even more the percentage of bots in internet traffic 🤖

Previously, I built a bot to automate posts on Bluesky, which is completely free and open-source. While the API is well-documented, it’s slightly harder than the X one.

In this piece, I share a step-by-step tutorial on automating posts on X. You’ll learn how I built a bot that publishes five parenting-related posts per day, using articles scraped from Today’s Parent, and AI images generated using Nano Banana.

Let’s get started!

Create a bot account on X

To get started, you’ll need two X accounts: your main account and a separate one for the bot. I’ll assume you’re already familiar with creating an account on X. The process is identical for the bot, with just one small difference.

The configuration is simple, just open your bot account, head to Settings, then Account Information, and choose Automation. From there, pick your managing account, usually your main one.

Once done, you should be able to see something like “Automated by @dadhalfdev” in your bot’s profile page.

And that’s all to create your bot. Now, we need to give it the right permissions to post.

Go to the X Developer Portal page of your bot’s account. There you’ll find the settings to configure your project.

On the settings tab, click on Edit to give your bot the permissions to publish.

⚠️ Note: Make sure you edit your project before using the keys, otherwise, you’ll need to regenerate them.

Once clicked, you’ll see the User authentication settings page. Here, in order to post, you need to check the radio button: Read and write and Direct message.

For the type of app, choose the one that includes bot. For the app info, you can use localhost.com for both the Callback URI and Website URL in this tutorial.

Back on the Settings page, you’ll see the Keys and tokens page. There, you will need to generate or regenerate and save them for the API call. To post, you’ll need the following keys and tokens:

• Bearer Token: For request authentication.

• API Key (Consumer Key): Username that represents your App when making API requests.

• API Secret (Consumer Secret): Password that represents your App when making API requests.

• Access Token: Token for OAuth 1.0.a API requests.

• Access Token Secret: Secret for OAuth 1.0.a API requests.

In this section, we’ve covered how to create an X bot and give it the right permissions. We’re now going to see how to use the credentials above with Tweepy, to seamlessly post on X.

Post on X with Python

If you visit the X API V2 official documentation. You’ll find a setup that can be hard for those who only want to post or retrieve information. The approach used in that documentation uses the Flask framework, but in this tutorial, we’re going to use Tweepy, which makes everything way easier.

First, you need to install Tweepy with this pip command:

pip install tweepy

And here’s the script to make a simple post on X:

import os
import tweepy
import dotenv
dotenv.load_dotenv()

### AUTHENTICATION ###

# Take these from the Developer Portal
ACCESS_KEY = os.getenv("ACCESS_KEY")
ACCESS_SECRET = os.getenv("ACCESS_SECRET")
CONSUMER_KEY = os.getenv("CONSUMER_KEY")
CONSUMER_SECRET = os.getenv("CONSUMER_SECRET")
BEARER_TOKEN = os.getenv("BEARER_TOKEN")

# Authenticate to X
auth = tweepy.OAuthHandler(
    CONSUMER_KEY, CONSUMER_SECRET, ACCESS_KEY, ACCESS_SECRET)

api = tweepy.API(auth)
try:
    api.verify_credentials()
    print("Authentication OK")
except tweepy.TweepyException as e:
    print(f"Error during authentication: {e}")

### POST IF AUTHENTICATION IS OK ###

client = tweepy.Client(
    consumer_key=CONSUMER_KEY,
    consumer_secret=CONSUMER_SECRET,
    access_token=ACCESS_KEY,
    access_token_secret=ACCESS_SECRET,
)

post = "Hey I'm the Bot Half Dad! Happy to share the latest news about parenting and family life."

response = client.create_tweet(text=post)
print(f"Tweet posted successfully: {post}")
print(f"Tweet posted! ID: {response.data['id']}")

If you want to add an image or a video to your post, you need these extra lines:

img = api.media_upload("your_img.png")
response = client.create_tweet(text=post, media_ids=[img.media_id])

With the code snippets above, you’re ready to start posting from your machine, and you can do it up to 500 times per month with the free tier.

Now that you have the pillars to build your bot, a few key questions arise:

1. What should you post?

2. How do you automate the posts?

3. How do you create engaging content if you’re not the one writing it?

4. How do you grab attention with the right images or videos?

I will start by tackling the first three questions above. Then, for the fourth, I’ll show you how to use the Gemini API to make some super cool AI images with Nano Banana.

Extract the data, summarize, and automate the posts

Before creating a bot, it’s important to clarify your goals. Do you want to generate income? Drive traffic to your website? Share tech insights? Post political content? Or simply do it for fun?

For instance, I know that my parenting news bot will be difficult to drive engagement on a social network like X. But if you’re into politics, tech, or cryptocurrencies, you might get a lot of attention and therefore redirect traffic to your personal website or blog.

My X bot uses a web scraper that runs daily to extract the newest blog posts about parenting from the Today’s Parent website. You can find the web scraper on Apify:

But I also have scrapers for tech and AI news:

• TechCrunch Scraper

• Hackernoon Scraper

For cryptocurrencies, blockchain, and web3:

• Cointelegraph Scraper

• Decrypt Scraper

These are just some examples of data crawlers you can use for your bot, but you can find many more on the Apify store.

Once you have the data your bot will use, you have several options. In my case, I use an LLM to summarize a randomly selected article from my dataset and then post the summary along with the article’s link.

The LLM I use to create the post previews is DeepSeek V3, which you can use for free with OpenRouter.

Let me show you how to use any LLM with the OpenAI API to summarize your data or perform other tasks.

First, you need to install the OpenAI Python library:

pip install openai

And here’s the full Python script:

from openai import OpenAI
import pandas as pd

def create_text_for_post(row_index):
    """ Create a text for post based on daily sample row index """

    # Read the daily sample csv file
    df = pd.read_csv('daily_sample.csv')

    # Get first row author X
    author_twitter = df.iloc[row_index]['author_twitter']

    # Check if author_twitter is not NaN and is a string
    if pd.notna(author_twitter) and isinstance(author_twitter, str):
        logger.info(f"Original author_twitter: {author_twitter}")
        if ".com/" in author_twitter:
            author_twitter = "@" + author_twitter.split(".com/")[1]
            logger.info(f"Processed author_twitter: {author_twitter}")
    else:
        logger.info("author_twitter is NaN or not a string, setting to None")
        author_twitter = None

    # Get first row link
    link = df.iloc[row_index]['link']

    # Get first row title
    title = df.iloc[row_index]['title']

    # Get first row content
    content = df.iloc[row_index]['content']

    # Get first row category
    category = df.iloc[row_index]['category']

    # Get first row author display name
    author_display_name = df.iloc[row_index]['author_display_name']

    # Create and format the prompt
    template = PromptTemplate(
        input_variables=[
            "content",
            "link",
            "category",
            "title",
            "author_twitter",
            "author_display_name"],
        template=PROMPT_TEMPLATE_POST,
    )

    prompt = template.format(
        content=content,
        link=link,
        title=title,
        author_twitter=author_twitter,
        author_display_name=author_display_name,
        category=category,
    )

    client = OpenAI(
        base_url="https://openrouter.ai/api/v1",
        api_key="<your_openrouter_key",
        )

    completion = client.chat.completions.create(
        model="deepseek/deepseek-chat-v3.1:free",
        messages=[
            {
            "role": "user",
            "content": prompt
            }
        ]
    )

    response_text = completion.choices[0].message.content
    logger.info(f"Response text: {response_text}")
    return response_text

Don’t bother too much about understanding the first lines of the Python function. That’s basically my way of tackling the dataset I’ve scraped.

From the dataset’s row, I get the features that are important for the LLM’s prompt template. They are: the link, the title, the category, the content, the author’s name, and the author’s X/twitter nickname.

Here’s the prompt I use to create the text for my post, based on the inputs:

PROMPT_TEMPLATE_POST = """
    You are a specialist in parenting news about {category}. You're also 
    a X-posting expert and influencer. You're tasked with writing a tweet 
    about the following:

    ARTICLE TITLE: {title}
    ARTICLE AUTHOR NAME: {author_display_name}
    ARTICLE AUTHOR X/TWITTER: {author_twitter}
    ARTICLE CONTENT: {content}

    INSTRUCTIONS:
    1. Analyze the provided {content} and {title} and summarize the article 
    in a post with a max length of 280 characters. Note that this is the max 
    length and you need to leave some space for the {link} 
    and {author_twitter} or {author_display_name}. 
    So make sure that the total lenght has the summary, 
    the {link} and the {author_twitter} or {author_display_name}. Keep short 
    sentences and separate them with an empty line in between.
    2. If {author_twitter} is an empty string ("") or None value, 
    use {author_display_name} instead to mention the author of the article. 
    But always prioritize {author_twitter} first.
    3. Structure the post with emojis, but don't use a lot of them, and they
    should match the {content} or {title}.
    4. Sound like a human and mostly a parent.
    You're providing parenting information and sometimes dads and moms are 
    exhausted so they need a breeze of fresh air, and some humor whenever 
    the {content} or {title} allows it.
    5. You can start by mentioning {author_twitter} or {author_display_name},
    something like "@CourtneyLeiva recently wrote", but you can also start 
    with the summary of the content and finalize with something 
    like " - by @CourtneyLeiva".
    6. The {link} is always after the summary of the {content} and the 
    {author_twitter} or {author_display_name}. And leave a an empty line 
    before the {link}.
    """

The output generated with this function is sent to the previous script I’ve shared with Tweepy to publish on X.

This is how the output looks on the app:

What about the image in this post? We’ll get there in the next bonus section.

If there were no image, the post would display the link’s image preview below the summary.

Tagging authors with their X nicknames can also be a smart way to catch their attention and boost engagement with your bot.

Before explaining to you how I created the cute image, you need to know how to automate the posts. Most likely, you don’t want to run it from your local machine and leave it turned on day and night.

That’s why I run the scripts on my Contabo VPS by configuring a cron job that triggers daily at a specified time.

I’m currently using the Cloud VPS 30 for only €12.5/month, which has 24 GB RAM (more than my laptop), 8 vCPU Cores, and 400 GB SSD. This plan is more than enough for most automation tasks, but feel free to have a look at the other VPS and VDS options available.

These are the cron jobs I run on my Contabo VPS to extract and publish:

# PUBLISH ON X
00 02 * * * cd ~/projects/bothalfdad && /usr/bin/python3 main.py 0
00 05 * * * cd ~/projects/bothalfdad && /usr/bin/python3 main.py 1
00 11 * * * cd ~/projects/bothalfdad && /usr/bin/python3 main.py 2
00 15 * * * cd ~/projects/bothalfdad && /usr/bin/python3 main.py 3
00 20 * * * cd ~/projects/bothalfdad && /usr/bin/python3 main.py 4

# RUN TODAYS PARENT SCRAPER
30 00 * * * cd ~/projects/todays_parent_scraper && /usr/bin/python3 src/__main__.py "Pregnancy"
35 00 * * * cd ~/projects/todays_parent_scraper && /usr/bin/python3 src/__main__.py "Kids"
45 00 * * * cd ~/projects/todays_parent_scraper && /usr/bin/python3 src/__main__.py "Baby"
55 00 * * * cd ~/projects/todays_parent_scraper && /usr/bin/python3 src/__main__.py "Family"
05 01 * * * cd ~/projects/todays_parent_scraper && /usr/bin/python3 src/__main__.py "Shopping"

I’ve decided to post 5 times per day, which is around 150 times per month, still far from the maximum of 500 posts/month with the free tier.

With all the information you have up to this point, you should be able to create your first bot on X. But what will differentiate your bot from the others?

Bonus: Create Nano Banana images for X posts with the Gemini API

In this section, I’ll show how to embed AI-generated Nano Banana images using the Gemini API.

First, you need to access Google Studio with your Google account.

In the left corner of Google Studio, you can see the option Get API key.

You’ll need to select a project with a billing plan, as using the API incurs a cost.

If you don’t have a project, you can follow the steps to create one on Google Cloud Platform using this insightful piece.

Also, you need to install the following Python packages:

pip install langchain-core google-genai

Here’s the function to create an image:

from langchain_core.prompts import PromptTemplate
import pandas as pd
from google import genai
from PIL import Image
from io import BytesIO

def create_image_for_post(row_index):
    """ Create an image for post based on daily sample row index """

    # Read the daily sample csv file
    df = pd.read_csv('daily_sample.csv')

    # Get first row title
    title = df.iloc[row_index]['title']

    # Get first row content
    content = df.iloc[row_index]['content']

    # Create and format the prompt
    template = PromptTemplate(
        input_variables=["content", "title"],
        template=PROMPT_TEMPLATE_IMAGE,
    )

    client = genai.Client(api_key="<google_studio_key>")

    response = client.models.generate_content(
        model="gemini-2.5-flash-image-preview",
        contents=[template.format(content=content, title=title)],
    )

    for part in response.candidates[0].content.parts:
        if part.text is not None:
            print(f"Response text: {part.text}")
        elif part.inline_data is not None:
            image = Image.open(BytesIO(part.inline_data.data))
            image.save("generated_image.png")    

Just like I did to create the post text, here I do something very similar using the same row index from my dataset.

In this case, I just use the article’s content and title for the prompt template:

PROMPT_TEMPLATE_IMAGE = """
Create an illustration in a bold comic book style, 
inspired by the {title} and {content} of the article. 

Requirements:
1. Use only the exact colors: #bae9f9, #ffeeaa, and #252525. 
   - #bae9f9 and #ffeeaa for backgrounds and highlights. 
   - #252525 for outlines and shading. 
2. Keep the composition clear and dynamic, like a comic panel. 
3. Allow at most 2–4 words of comic-style text 
(e.g., "Wow!", "Tip!", or a short caption). Avoid full sentences. 
4. Make the style playful and engaging for parents.
"""

This function is placed before publishing, so I can store the image first and then embed it in the post using Tweepy as initially demonstrated in this piece.

The pricing for the Gemini 2.5 Flash Image Preview model is the following:

With a total of 25 images, I’ve spent $1.33, so it can easily become expensive if you’re planning on posting a lot of images per day.

Nonetheless, the results are impressive:

Need help with automation, AI, or data science? Let’s talk.

Book a 20-min consultation

Conclusion

Creating a bot on X is simple, but making it stand out in a social media environment where multiple bots are running constantly is the hard part.

Fortunately, AI gives us more degrees of freedom to be creative and deliver information in the most engaging ways possible. Either it is through text, images, or video animations.

Tagging people or adding keywords that are trending on the platform is a good way to get followers, and once you get enough engagement, you can start thinking of strategies to generate income through it.

For instance, in my case, I can embed affiliate links of Montessori toys or promote family activities and baby products. In addition, I can also use it to get traffic to my personal parenting blog (I don’t have one by the way).

The options are endless, and now you have the starter kit to build yours!

If you’ve made a bot yourself, whether by following this tutorial or using other resources, feel free to drop it in the comments section. I’d love to see what you came up with!

I was a student of Engineering in Nanotechnology and Microelectronics, struggling to pay bills, going out, and traveling.

I used to live five minutes walking from Lisbon’s flea market, so I knew there was some potential to generate income there.

Maths and physics were consuming all my energy, so I needed something to disconnect. I found art as a relief, and I never thought I could monetize my work until I started to get very positive feedback from people.

So I tried, one Saturday morning, I packed my art pieces, which I called Drawnics. Took an old blue towel to put on the floor, and that was pretty much my stand.

To my surprise, people started buying these Drawnics, mostly tourists, so every week I was making more and more to sell at the flea market.

I built these characters from old Electronic Circuit Boards (PCBs) and old book covers. It was a way to upcycle electronic waste, and people loved the idea.

I made Drawnics until my graduation, then I started working, and I switched my creativity to other things.

I don’t remember the exact date I said goodbye to my old friends. But recently, with the editing power of Google’s Nano Banana and Veo3, the first thing that came to my mind was turning my Drawnics into animated characters.

And that’s what this article is about. I will guide you through all the steps I took to create my first video animation with Veo3, while mentioning the struggles and the mistakes I made, so you don’t have to.

Use Nano Banana for free with Google Gemini 2.5

If you search for Nano Banana on Google, in the first SERP results, you will find websites like nanobanana.ai, which provides a paid version of the model. My honest tip: don’t use it! You can get it for free, and many platforms are trying to scam users.

Simply go to Gemini and pick the tool Create images.

You can either create an image from scratch or upload one or multiple images and ask Gemini to edit them according to your prompt.

You can edit the images in a very natural way, and Gemini can more or less generate your idea, but if you want to go professional, I found a post on LinkedIn with several prompt templates you can try, like this one:

1. Photorealistic Scenes

For realistic images, think like a photographer. Mentioning camera angles, lens types, lighting, and fine details will guide the model toward a photorealistic result.

Template: A photorealistic [shot type] of [subject], [action or expression], set in [environment]. The scene is illuminated by [lighting description], creating a [mood] atmosphere. Captured with a [camera/lens details], emphasizing [key textures and details]. The image should be in a [aspect ratio] format.

Example Prompt: A photorealistic close-up portrait of an elderly Japanese ceramicist with deep, sun-etched wrinkles and a warm, knowing smile. He is carefully inspecting a freshly glazed tea bowl. The setting is his rustic, sun-drenched workshop. The scene is illuminated by soft, golden hour light streaming through a window, highlighting the fine texture of the clay. Captured with an 85mm portrait lens, resulting in a soft, blurred background (bokeh).

Here’s the link for the LinkedIn post.

My story started with a blue Drawnic, dressed in orange. Here’s the protagonist:

This is the prompt I used to turn it into a movie star:

Turn this guy into a real-life character. Make sure to preserve all the details. The environment should take place in a post-apocalyptic world. But make the surroundings with light a smooth colors, without too much noise, the focus is on the character.

The output:

As I started editing, I found some annoying issues that were hard to fix:

1. Pixar style

2. Aspect ratio

3. Hallucinations

Let me explain one by one in detail.

The Pixar style

By default, the model tends to create Pixar-style characters. I was trying to preserve the details of my drawings, but I couldn't get them in 3D. It makes sense in a way that the model was trained with many Pixar images, but struggles to create depth from pictures where it has no previous information.

The aspect ratio

When adding more than one image, Gemini may pick the aspect ratio of one of them.

A simple prompt like change to 16:9 aspect ratio does not work on Gemini. The workaround was to add an image with the right aspect ratio and tell the model to use it in the previous image.

I highly recommend that you use 16:9, as it will make your life easier at the moment of creating the videos with Flow from Google Labs. We will see this with more detail later in the article.

Hallucination

Again, when more images were added, I started labelling them. Something like character A, B, C, and so on. This worked until a certain point, then Gemini began generating characters from scratch or mixing the labels. The workaround was to avoid using the same chatbot instance for more than three or four requests.

In addition to the issues mentioned, sometimes Gemini fails to output an image, and you need to retry.

The more familiar you are with filmmaking vocabulary, the easier it becomes to generate the right takes. Regardless of the trials and the time I spent, in the end, I successfully created a collection of around 7 images to start making my video animation.

All images generated with Gemini 2.5 have watermarks, but you can easily remove them with platforms like this.

Use Veo3 for free with Google Gemini 2.5 Pro

Similar to Nano Banana, you may find scam websites in SERP results for Veo3. Such as Veo3.ai, now changing the name to SonoVid.ai. They are using the right model, but they charge more than simply using Gemini.

There are two main options to generate videos with Veo3. You can either do it using the Gemini chatbot or using Flow. The latter is better for storytelling and generating new videos from frames.

Before starting, you need to have a Google account and choose the Google AI Pro plan, which is free of charge for the first month. After the trial period, the price is €21,99/month.

The Pro plan gives you 1,000 credits, and each fast generation costs 20 credits, allowing for 50 generations per month. Generations prioritizing quality instead of speed cost 100 credits, leading to only 10 generations per month. In the following example, I’ve only used fast generations.

I will now start showing you the process of making my first video animation on Flow.

In Flow’s creator page, you can either select the option Text to Video or Frames to Video. In my case, I already had the images without a watermark ready to be animated, so I only used the Frames to Video option.

Make sure you select the Veo3 — Fast option and set the number of outputs to one, to avoid using too many credits.

Veo3 does not allow adding the start and ending frames, only one frame per video generation, so you really need to prepare the images in a way that does not compromise your video sequence. If you want to add a starting and ending frame, you need to specify Veo 2 instead, and it does not generate audio.

Once again, the more you know about filmmaking vocabulary, the better the results will be. I’m not really familiar with these terms, so I often use ChatGPT to help me out with the scenes.

Here’s the prompt for the first scene of my video:

The character advances toward the lens, each step deliberate. The camera retreats in a smooth, slow tracking move, keeping the character centered as if drawn toward us. The faint crunch of footsteps underfoot blends with a soft breeze, the ambient wind brushing against the silence, heightening the intimacy of the moment.

And here’s the scene:

You can see that Veo3 does a great job preserving the surroundings and the overall details of the image.

Unlike the Gemini watermark on the images, which is easy to remove, for the videos, I did not find a straightforward way of doing it. You would need to pay for Gemini’s ultra plan.

Continuing on the Flow creator’s page, you have the option to add the video to the scene builder, at the top left of your video generation: Add to scene.

After creating several shots, I moved to the scene builder to do some minor editing and optimize the transitions.

One of the best features of Flow is that it allows you to save a frame from a specific scene. This feature is great for transitions! Because you can create a new video from that frame, allowing you to extend the scene for more than 8 seconds. However, the audio can still be tricky to optimize.

When you create a new frame in Flow, you don’t need to worry about watermarks, because you’re already in Gemini 2.5 pro. Also, you shouldn’t face aspect ratio issues.

Though all my takes, I tried to keep the same post-apocalyptic feel, and a constant sound of wind in the background. Having the audio and the lip sync options on Veo3 is indeed great! But it also adds extra complexity to building a proper animation, where the characters must maintain the same voices and the audio does not differentiate too much from scene to scene.

Flow has limited editing capabilities. You can only trim parts of the scene. But if you’re happy with the output, you can download the full sequence.

In my case, I’ve decided to download videos one by one instead and use an external video editor.

Microsoft’s Climpchamp is a free tool that allows you to add transitions, text, and more. But if you want to go professional, I highly recommend Invideo online video editor.

Making a 1-minute video took me a few hours, but I had no previous experience in building animation videos. In total, I spent exactly 400 credits to make it. Here is the final work:

Need help with automation, AI, or data science? Let’s talk.

Book a 20-min consultation

Conclusion

The process of building a 1-minute animation video was long and costly. With only a few generations, I’ve already used almost half of the monthly credits.

If you decide to take the Gemini Pro version only because of Veo3, you need to realise that you won’t be able to make more than 3 short videos per month.

Therefore, I think this plan might be worth it to make simple 8-second animations, but not for filmmaking. Besides, you face the challenge of the watermark, which can be easily fixed with single videos, but not with a full animation.

Nonetheless, I’m really happy with the output. There’s a sense of nostalgia in seeing my old pieces of artwork come alive in the digital world. Characters that were once static, living only in my imagination or in the hands of those who bought them at the flea market, can now reach a wider audience, even if they’ve taken on a Pixar-like look.

Veo3 and Nano Banana make a powerful combo: Veo3 brings strong image-editing capabilities that breathe life into idle objects, while Nano Banana excels at preserving image details and generating natural motion and audio.

With that being said, maybe a Drawnics — Part 2 coming soon?

If you follow tutorials on YouTube, most of them are outdated, with components that are no longer used and a workflow editor that does not resemble the current one.

That’s ok, you can still learn from these videos, at least until you run into certain logical limitations. At that point, a tool designed to be intuitive and beginner-friendly can quickly turn into a real nightmare.

In this tutorial, I’ll show you how to build a chatbot that learns from two different Knowledge Bases (KBs). I’ll also point out the issues that cost me several hours of troubleshooting on Botpress’s Discord channel.

Let’s make Botpress a breeze to use!

1. Create a new agent (aka assistant)

In this example, we are going to use a PDF file about ETH (Ethereum’s currency) and a table with NFT collections as Knowledge Bases. The idea is to create a preview of what could be a powerful crypto and web3 agent, while showcasing key strategies to start building your first workflow.

Let’s start by creating an account on Botpress.

As soon as you create an account, it will guide you to configure an agent tailored for your needs. If you choose Customer Support, Brand Assistant, or Product Recommendation, a workflow will be generated, allowing you to skip some steps.

However, if you want to have full control over your workflow, I recommend using the Other option.

Following the Other option, you will be asked to add data to your Knowledge Base if you want to. You can add websites, files (.pdf, .doc, .html, and more), or simple text.

You don’t need to worry about chunking, indexing, or data preprocessing like you would when coding. The platform handles all of that for you. In this tutorial, we are going to add the files later in the studio page.

In the case you have already selected your KBs and an agent type like Customer Support, you would have at this point a chatbot ready to share or embed in your website (with limitations).

In our case, we have a pretty dumb workflow, so let’s start working on it by clicking Advanced!

2. Add Knowledge Bases (KBs) and Tables

In this tutorial, we are going to make the LLM (Large Language Model) query data from tables and a PDF file.

On the left side, you’ll find an icon for tables. Click on it and create a new table.

If you already have a CSV file and don’t want to manually input all the fields into the table, you can use the import option.

Here, I’m going to add a CSV file that was generated using my Magin Eden web scraper. It scrapes the latest information about NFT collections across multiple blockchains and Layer 2 chains.

Here’s a preview of the data I’m adding to the Botpress table:

Once the table is filled, we can move to the KB section. The icon is also on the left side of the studio page.

There is a default KB, which you can rename to something that illustrates your table data. Now, click on the tables icon and choose your table. Mine is called NFTCollectionTable.

Now, let’s create a second KB called ETH Info. I’m going to add a simple PDF file that compares the ETH (Ethereum’s blockchain asset) to oil. You can find this file here, perhaps you can even add the link to the KB instead of uploading the document.

3. Build a ready-to-go workflow that works for you

Now comes the amusing part. Building the flow of the chatbot!

On Botpress, the workflows are built using Nodes and Cards. You can see the Nodes as containers that are made of several Cards. The latter are mostly actions and transitions. We will take a look at several card examples in this guide.

You can find the workflows icon on the left side of the studio page. The default example can be messy, and for a good follow-up to this tutorial, I recommend you remove all the nodes by simply clicking on them and pressing Delete.

One thing that I realised when building my first Botpress agent is that when you launch the chatbot, it will always wait for the user's input to start a conversation.

In most cases, that’s fine, but what if you want the chatbot to introduce itself and give a short preview of what it is capable of?

How does the bot start talking before you do?

After spending some time on YouTube and Botpress’s Discord channel, I found the solution to this issue. The only thing you need to do is start the workflow from a trigger event called Conversation Started, instead of the default Start node.

While this solution works, there’s still a small delay when you open the chatbot before triggering the intro message.

Add an introduction text and a multiple-choice node

Since we want our chatbot to start the conversation before the user’s input. It would be nice to make a short and funny introduction to break the ice.

Right-click on your mouse to create a node, then insert the card Text. Write something like:

“Hey, I’m the crypto bro, who knows more than all crypto bros combined!”

Up to now, there’s no interaction between the user and the agent. Let’s add a Multiple Choice card in the same node, just below the Text card.

This card is also easy to configure, all you need is to make a question and present several options.

In our example, I want to create two paths (two choices), one that is related to NFT collections and another for the Ethereum PDF information.

If you click on publish, in the right corner of the studio page, you’ll get a shareable link, where you can see what you’ve done so far. Otherwise, you can also test it internally by clicking Test your bot on the emulator window.

Add autonomous nodes and exit them

Good, now we have everything settled to put the LLMs to work for us with the Autonomous Nodes. Simply press the right-click on your mouse and select Autonomous Node. Let’s start with the NFT collections.

In the image above, you can see I’ve added the Autonomous Node with two other cards:

1. Search Knowledge Base: This ensures that the LLM queries all Knowledge Bases or just a specific one. In this case, I’ve set it up to only learn from the NFT collections one.

2. Transition (“Ending”): This Transition card allows for exiting the autonomous node based on some conditions written in the prompt.

This is how the prompt looks:

You are an expert in NFT collections, with access to detailed data from the “NFT Collections” knowledge base. Your goal is to guide the user toward choosing the NFT collection that best fits their needs.

1. Start by greeting the user warmly and establishing trust.
2. Ask targeted questions to understand their preferences and goals before showing any collections:
 — What is your budget range?
 — Do you prefer a specific blockchain (e.g., Solana, Ethereum, Polygon)?
 — Looking for a scpecific volume or volume change (%)?

3. Use their answers to filter the knowledge base and present a short, tailored list of 3–5 NFT collections with:
 — Collection name
 — Blockchain
—Key parameters from Knowledge Base

4. If the user says something like:
 — “I want to see more collections”
 — “Where can I get more data?”
 — “Show me more options”
or expresses any desire for additional info,
respond with:
**”You can explore even more NFT collections here: https://apify.com/dadhalfdev/magic-eden-nft-collections-scraper?fpr=ulosg"**

5. If the user seems satisfied or the conversation is coming to a natural close, transition to the “Ending” card below.

Make sure your tone is friendly, knowledgeable, and approachable. Avoid jargon unless the user is clearly an experienced NFT trader.

⚠️ Important Note

Point number 5 is super important! It’s a fix for an issue that I saw many people struggling with (myself included), which is exiting the Autonomous Node. Most likely, you don’t want to speak with the assistant indefinitely, you want to be able to exit when the conversation ends or a specific keyword is mentioned.

The way I saw to solve this is by mentioning the Transition card name in the prompt itself.

According to the prompt, the chatbot should be able to provide the right NFT collections based on the user's inputs.

When launched, it indeed does that, and even better, it takes the images from the tables and puts the best matches in a horizontal carousel view 🤩

Now we need a second Autonomous Node for the I’m an ETH maximalist option.

In this case, we want to use the KB based on the PDF file we’ve previously uploaded and also create a prompt tailored for this option. In addition, I want to add an affiliate link in the prompt in case the user wants to buy ETH.

Below is how the ETH branch looks in the chatbot after adding the Autonomous Node.

We can see that it is indeed querying the right KB, and as intended, showing the affiliate link!

Do you need help with automation, web scraping, AI, data, or anything that my laptop can deliver? Feel free to reach me on Upwork! 👨‍💻

Let's build together!

The final workflow looks like this:

Need help with automation, AI, or data science? Let’s talk.

Book a 20-min consultation

Conclusion

The final workflow is quite simple, it only contains 4 nodes. But the goal was to guide you through the full process of creating an agent, while highlighting the solutions for two common issues:

1. “How does the chatbot start the conversation before the user’s input”.

2. “How to exit the Autonomous Node”.

In the example provided, I used the Conversation Started trigger instead of the Start node at the beginning of the workflow to address the first issue. And I call the Transition Ending in the prompts, in order to exit the nodes.

More transitions could be added, for instance, if the user wants to search the web about a certain NFT collection, we could exit the node on that intent and connect another node with a Web Search card.

Besides, I also wanted to show you how easy it is to do affiliate marketing. If you have a Knowledge Base about electronic or cosmetic products, you can contact providers and add affiliate links while the user engages in a conversation with your chatbot.

While Python is one of the top programming languages for data analysis and data visualization, there are still few libraries and frameworks capable of delivering production-ready dashboards.

Libraries like Seaborn, Matplotlib, and Bokeh, are great for exploratory data analysis, building reports, and even Graphical User Interfaces (GUIs), but they don’t natively support HTML, CSS, and JavaScript. Therefore, they are not suitable for web applications.

Solutions such as Gradio and Taipy were built to facilitate Data Scientists and Machine Learning (ML) engineers building Proof of Concepts, Demos, and simple interfaces, where the main focus is to test a product, rather than making it ready for production.

Let’s now have a look at the top framework to build insightful and appealing dashboards and its limitations.

The best framework for data visualization

The top framework to build advanced dashboards, with an extensive list of components and flexibility of customization, is Dash Plotly.

This framework is fully open-source and it’s available for both Python and JavaScript, which makes it widely used, with many examples available on GitHub and forums. This vast publicly available data made Large Language Models proficient in building Plotly dashboards, which eases dashboard creation through vibe coding.

Standard Dash Plotly components are built with React, and they can be customized in order to look more appealing and suitable for your needs. There’s even a React tutorial for Python developers, so you can start building without requiring learning React from scratch.

The fast approach

But let’s say you don’t want to spend countless hours customizing your components. In that case, you can use open-source libraries such as Dash Bootstrap Components and many more, which you can find here.

Among the several examples on the list above, there’s one that will make your dashboards absolutely beautiful!

That’s Dash Mantime Components, with over 90 components from the React Mantine library.

Compared to other Dash Plotly libraries, DMC offers a modern look, with easy customization, requiring less HTML and CSS knowledge.

First steps with Dash Mantine library

One good practice for building dashboards in Python is to first create a Flask server. This gives you more flexibility in case you want to add pages that are not built using Dash or add extra parameters to the server.

Let’s also install Dash and DMC.

pip install flask dash dash-mantine-components

Once installed, we can start building a simple dashboard. Let’s create a login page:

from dash import Dash, _dash_renderer
from dash import dcc, html
import dash_mantine_components as dmc
from flask import Flask

# Flask server
server = Flask(__name__)
server.secret_key = 'parenting'

# Set the react version for DMC components
_dash_renderer._set_react_version("18.2.0")

# Stylesheets
external_stylesheets = [
    dmc.styles.ALL,
    dmc.styles.CHARTS
]

# Set up a single Dash app with multiple pages
app = Dash(
    name='parenting_dashboard',
    server=server,
    title='Parenting Dashboard',
    external_stylesheets=external_stylesheets,
    assets_folder='assets')

# Define the layout for the different pages
app.layout = dmc.MantineProvider([
    dmc.LoadingOverlay(
        id="loading-overlay-login",
        visible=True,
        loaderProps={
            "color": "#f08b56",
            "size": "md"
        },
        style={
            "background": "white",
            "opacity": 1,
        }
    ),
    dcc.Location(id='url', refresh=False),
    dcc.Store(id='store', storage_type='session'),
    dcc.Store(id='current-page', storage_type='session'),
    html.Div(
        dmc.Stack([
            dmc.TextInput(
                id="username",
                label="Username",
                size="lg",
                miw=300,
                placeholder="Enter your username",
                required=True,
            ),
            dmc.PasswordInput(
                id="password",
                label="Password",
                size="lg",
                miw=300,
                placeholder="Enter your password",
                required=True,
            ),
            dmc.Button(
                "Login",
                id="login-button",
                size="lg",
                miw=300,
                color="#f08b56"
            ),
            dmc.Alert(
                "Invalid credentials",
                title="Error",
                hide=True,
                id="alert-login",
                color="red",
                withCloseButton=True,
            ),
        ],
            justify="center",
            align="center",
            style={
                "width": "100%",
        },
            className="main-container-login"
        ),
        id="main-container",
    ),
])

if __name__ == '__main__':
    # Run the server
    server.run(host='0.0.0.0', port=8080, debug=True)

Overall, this main.py code snippet does the following:

1. Creates a Flask server.

2. Sets the React version to 18.2.0 using _dash_renderer.

3. Adds DMC styles. You should be able to get all styles with dmc.styles.ALL, but if not, you can add them individually like dmc.styles.Charts.

4. Instantiates the dash application using Dash, where a name and a title are given, along with the Flask server, the stylesheets, and the assets folder that contains the CSS files.

5. Wraps the login page inside the dmc.MantineProvider object, which is key to rendering the DMC library components.

6. Finally, launches the Flask server on port 8080.

This first page looks like this:

Callbacks and other DMC components

The previous login page requires a username and a password. Otherwise, it prompts an alert message.

Let’s take a look at the callback function that handles the authentication.

from dash.dependencies import Output, Input, State, ALL
from dash import no_update, clientside_callback
from app.components import Login
from app.settings import ParentingSettings


class Callbacks:
    """
    Class to handle main callbacks for the app.
    """

    @staticmethod
    def callback_login(app):
        """
        Callback function to handle the login.
        """
        @app.callback(
            Output('store', 'data'),
            Output('alert-login', 'hide'),
            Output('login-button', 'n_clicks'),
            Input('login-button', 'n_clicks'),
            State('username', 'value'),
            State('password', 'value'),
        )
        def login(n_clicks, username, password):
            if n_clicks is None:
                return no_update, True, 0
            if n_clicks > 0:
                if username == 'toddler' and password == 'parent123':
                    return 'toddler', True, 0
                elif username == 'preschooler' and password == 'parent456':
                    return 'preschooler', True, 0
                else:
                    return no_update, False, 0

By using the ids previously defined in the layout, we can create inputs and outputs. The callback function above takes the username, the password, and the button as inputs. The State is a type of Input, that uses its values but does not require an action. This avoids triggering the callback if the user fills in the dmc.TextInput or the dmc.PasswordInput.

The outputs of this callback are simply the hidden value for the alert component and store value. The latter will be saved in the session of the browser, and based on it, you can generate different layouts. This is important when handling different clients.

@staticmethod
def callback_store(app):
    """
    Callback function to handle the store value.
    """
    @app.callback(
        Output('main-container', 'children'),
        Output('loading-overlay-login', 'visible', allow_duplicate=True),
        Input('store', 'data'),
        prevent_initial_call=True,
    )
    def parenting_category(category):
        if category == 'toddler':
            main_container = Store.gen_navbar(category)
            return main_container, False
        if category == 'preschooler':
            main_container = Store.gen_navbar(category)
            return main_container, False
        else:
            return Login.gen_login_form(), False

Once logged in, the session value triggers the callback function above, and a layout is generated accordingly.

⚠️Note: Each new callback function should be added to the main.py function to be triggered.

Build a navbar with DMC

After the authentication, a new layout is shown inside the dmc.MantineProvider. Let’s have a look at a common piece on a dashboard, a navbar with selectors to control the charts.

import dash_mantine_components as dmc
from dash_iconify import DashIconify

def gen_navbar(category):
    """
    Generate the navbar based on the parenting category.

    Args:
        category (str): The parenting category to
        generate the navbar for (eg. "toddler", "preschooler")

    Returns:
        html.Div: The navbar for the parenting category.
    """

    max_date = datetime.now()
    min_date = datetime(2024, 1, 1)

    settings = ParentingSettings(category)

    return \
        html.Div([
            html.Div([
                dmc.Stack([
                    dmc.DatePickerInput(
                        id="start-date-input",
                        value=max_date - timedelta(days=30),
                        label="Start Date",
                        className="nav-item",
                        required=True,
                        minDate=min_date,
                    ),
                    dmc.DatePickerInput(
                        id="end-date-input",
                        value=max_date,
                        label="End Date",
                        className="nav-item",
                        required=True,
                        maxDate=max_date,
                    ),
                    dmc.Select(
                        id="activity-selector",
                        leftSection=DashIconify(
                            icon="mdi:baby-face"),
                        data=settings.properties_selector,
                        label="Select Activity Type",
                        searchable=True,
                        value=settings.properties_selector[0]['value'],
                        className="nav-item"
                    ),
                    dmc.Select(
                        id="pages-dropdown",
                        leftSection=DashIconify(
                            icon="material-symbols:bar-chart"),
                        searchable=True,
                        label="Dashboard Page",
                        comboboxProps={
                            "transitionProps": {
                                "transition": "pop", "duration": 200}},
                        data=settings.pages_selector,
                        value='dialogs',
                        className="nav-item",
                    ),
                    dmc.RadioGroup(
                        children=dmc.Group(
                            [dmc.Radio(l, value=k, color=c, id=d) for
                             k, l, c, d
                             in settings.timeframes_radio_options],
                            my=10),
                        id="radiogroup-timeframes",
                        value="week",
                        label="Select a Timeframe",
                        size="sm",
                        mb=10,
                        className="nav-item"
                    ),
                    dmc.Space(h=20),
                    html.Div([
                        dmc.Button(
                            "Sign Out",
                            id="log-out-button",
                            color="#f08b56",
                            leftSection=DashIconify(
                                icon="material-symbols:logout"),
                            size="xs",
                            n_clicks=0,
                        )
                    ], className="nav-item", style={"margin": "auto"}),
                ])
            ], className="navbar-col"),
            html.Div(id="page-content")],
            id="main-container",
            style={
                "display": "flex",
                "minHeight": "100vh",
                "width": "100%",
        }
        )

The layout above is controlled by the store value (category), which in our example can either be toddler or preschooler. This category is then used to create specific components and logic for the dashboard.

Now back to the DMC elements, the dmc.Stack component is useful to align the items vertically, while the dmc.Group is used for horizontal alignment.

These are the controlling widgets being used in the navbar:

• dmc.DatePickerInput() used for picking dates. You can control the minimum and maximum dates.

• dmc.Select(), a dropdown element. Takes the data input as a dictionary.

• dmc.RadioGroup(), a group of radio buttons. You can customize them individually with dmc.Radio().

• dmc.Button(), a simple button.

This is how the navbar looks like:

Did you see the cool icons? That’s possible with the DashIconify object. You can install this library with pip:

pip install dash-iconify

Once installed, you can integrate the DashIconify into your control widgets and select icons from a library with thousands of options. Find them here.

Pagination with DMC

We can now have a look at the callback function that powers the pagination of the dashboard using the following data for the page selector:

[
  {"label": "Daily Activities", "value": "daily"},
  {"label": "Sleep Patterns", "value": "sleep"},
  {"label": "Meal Tracking", "value": "meal"},
  {"label": "Development Milestones", "value": "development"},
  {"label": "Health Issues", "value": "health"},
  {"label": "Behavior Notes", "value": "behavior"},
 ]

In brief, we aim to render a page on the left side of the navbar, based on the selected value. This page layout will be displayed in the id page-content and it can show charts, tables, and overall data insights.

Let’s have a quick look at the callback logic:

  @staticmethod
  def callback_page_layout(app):
      """
      Callback function to open the layout according to the dashboard input.
      """
      @app.callback(
          Output('page-content', 'children'),
          Output('url', 'pathname'),
          Input('store', 'data'),
          Input('pages-dropdown', 'value'),
      )
      def display_page(category: str, page: str):

          if page == 'daily':
              return gen_daily_layout(category), '/'
          elif page == 'sleep':
              return gen_sleep_layout(category), '/sleep',
          elif page == 'meal':
              return gen_meal_layout(category), '/meal'
          elif page == 'development':
              return gen_development_layout(category), '/development',
          elif page == 'health':
              return gen_health_layout(category), '/health',
          elif page == 'behavior':
              return gen_behavior_layout(category), '/behavior',

The function above takes a page dropdown value as input, as well as the stored value category, and outputs a layout on page-content while redirecting to the target URL pathname.

This logic creates something like this:

The idea is that the navbar is fixed, and only the Div element on the left (page-content), changes every time the user switches pages.

Create charts with DMC

Cool, so we finally start to see the dashboard flourishing! Let’s have a look at some DMC charts and how they differ from the standard Dash Plotly ones.

The bar chart we saw above was built using the following function:

from dash import html, dcc
import dash_mantine_components as dmc
from app.settings import ParentingSettings
from dash_iconify import DashIconify
from app.components import General
from datetime import datetime, timedelta 


@staticmethod
 def gen_daily_layout(category):
      """
      Generate the layout for the daily activities page.
      """

      settings = ParentingSettings(category)

      return html.Div([
          dmc.Title(
              "Daily Activities Tracking",
              className="dashboard-page-title",
              order=1
          ),
          dmc.Space(h=40),
          dmc.Stack([
              General.gen_loading("loading-overlay-daily"),
              dmc.Group([
                  General.gen_paper_box(
                      "Average Activities", "avg-daily-text"),
                  General.gen_paper_box(
                      "Total Activities", "total-daily-text"),
              ], gap=50),
              dmc.Space(h=20),
              dmc.Paper([
                  dmc.BarChart(
                      id="bar-chart-daily",
                      withLegend=True,
                      maw=1000,
                      h=400,
                      data=[],
                      withTooltip=True,
                      dataKey="weekYear",
                      type="stacked",
                      xAxisLabel="Weeks",
                      yAxisLabel="Activities",
                      yAxisProps={
                              "domain": [0, "dataMax"],
                              "allowDataOverflow": True,
                      },
                      style={"width": "100%"},
                      series=settings.properties_default_series
                  ),
                  General.gen_download_button("download-button-daily",
                                              margin_left='auto'),
                  dcc.Download(id="download-data-daily"),
              ],
                  style={"width": "100%"},
                  maw=1000, radius="sm", p="lg", shadow="sm"),
          ], align="center"),
      ], className="dashboard-page-content")

In the script above, the object that creates the bar chart is dmc.BarChart and takes a few inputs to generate a modern-looking dynamic plot.

Standard libraries require a lot of customization to prettify the charts. Otherwise, they look like this:

Not amazing, right? This is the DMC version with almost no customization:

More examples:

These are just some examples. We could go on and explore even further! You can try to build other types of charts and play with callback functions by following the DMC documentation.

Need help with automation, AI, or data science? Let’s talk.

Book a 20-min consultation

Conclusion

While the Dash Mantine Components library offers great-looking charts and widgets compared to standard Plotly libraries, it still has some limitations.

For instance, if you want to create a funnel chart, you won’t find an object in DMC for it. You would need to stick with Plotly Express or Graph Objects.

This can be a problem, mostly if all your elements are DMC and suddenly some of them have a different, less modern look.

Nonetheless, the DMC repository is still active on GitHub, last commit 4 days ago at the time I’m writing, meaning that more components may be added soon.

Another important point of this library compared to using traditional Plotly components is that it is not great for vibe coders. I’m a Cursor user, and I found it hard to create DMC elements using the composer.

Therefore, take your time, read the documentation, and start building production-ready dashboards with Python.

With Apify, you can run and upload web scraping scripts, which are called actors. You can build them with either JavaScript or Python programming language, and Apify provides rich documentation in both languages.

But why should you use Apify instead of building your API platform and integrating your web scraping scripts?

1. Well-known platform: With this marketplace, you can reach a wide audience due to its position in the market. Therefore, you don’t need to spend much energy and money promoting your platform. If you have a fast script, which comes with fair prices and there’s not enough competition, you may be able to monetize it sooner than you thought.

2. No need for infrastructure: If you want to create your web scraping platform, that comes with costs. You may need to host your script on a VPS and pay a monthly fee, as well as buy a domain for your API endpoint. Not to mention the costs associated with building a website and database. Apify handles the hosting, scaling, and execution of the scripts, saving costs and effort.

3. Two different types of monetization: You can monetize your actor either with a monthly subscription or a fixed amount for 1,000 requests. The process is extremely simple, all you need is to do is provide your details and associate your PayPal account to start getting paid.

4. Ensure reliability: Apify has some quality standards that need to be respected, such as your actor shouldn’t take more than 5 minutes to give results. It also provides logs, error reporting, and a robust infrastructure to debug and optimize your script to make it the leading option in the market.

5. Focus on Development: Due to the robust deployment infrastructure, you don’t need to worry about DevOps and other engineering domains that are required when building your platform. This is great for developers who can solely focus on making a high-quality web scraping actor.

The examples mentioned above are just a few. Apify offers additional benefits, such as reliable customer support and insights into your script’s issues and performance.

While it offers all these perks, diving into the full SDK documentation or trying to grasp the Apify workflow can consume valuable time. That’s why I’ve decided to write this article and provide a straightforward tutorial to help you seamlessly migrate your web scraping script to an Apify actor and begin monetizing your work.

How to create an Apify actor

Before deploying an Apify actor, the first step is to have a functional web scraping script. In this guide, I’ll walk you through the key components of my Polymarket web scraping code, which have been adapted to work with the Apify SDK.

There are two main Apify packages for Python, so make sure not to mix both. They are:

• Apify Python Client: The official library for accessing the Apify API, offering features such as automatic retries and convenient utility functions. All requests, responses, and errors are handled in JSON format with UTF-8 encoding. Requires Python 3.8 or higher.

• Apify SDK for Python: The library used for building actors on the Apify platform.

To install the Apify SDK for Python via pip:

pip install apify

You also need to install the Apify CLI to interact and push your code to the cloud.

npm install -g apify-cli

Once the CLI is installed, you must create an account on the Apify website.

Now you can use the CLI to log in from your local machine:

apify login

This will ask you to authenticate through a token, which you can find in your profile settings or by directly connecting to the console in your browser.

At this point, we can already initialize the repository, even if it does not yet contain the web scraping logic.

apify init

This will create the storage and .actor folders, along with a .gitignore file.

Let’s begin creating our actor by using a default web scraping script as the base. We’ll distinguish between the Actor Script and the Default Script in the following steps.

To make the process easier, let’s create the logic in two main files inside a src folder, which are the main.py and the __main__.py. Overall, the tree of the Actor Script repository should have the following shape:

├── .actor
│   ├── actor.json
│   ├── Dockerfile
│   ├── input_schema.json
├── src
│   ├── __main__.py
│   ├── main.py
├── storage
├── README.md
├── requirements.txt

Step 1 — Create a __main__.py script

In the Default Script of the Polymarket web scraper, I have the following code for the __main__.py inside the src folder.

""" This script launches the Polymarket Scraper"""
# pylint: disable=E0402
import time
import asyncio
import logging
from rich.logging import RichHandler
from main import main

SEARCH_TYPE = "New"
NUM_MARKETS = 40


if __name__ == "__main__":
    # Logging configurations
    logging.basicConfig(
        filename="logs/output.log",
        filemode='w',  # This ensures the file is overwritten each time
        format='%(asctime)s | %(levelname)s: %(message)s',
        level=logging.DEBUG)
    logger = logging.getLogger()

    # Create a RichHandler for nice formatting
    rich_handler = RichHandler()
    logger.addHandler(rich_handler)

    asyncio.run(main(logger, SEARCH_TYPE, NUM_MARKETS))

So you can see that in this file I created a logger object and I run the main scraping function asynchronously, which takes the logger, a search type, and the number of markets to scrape as inputs.

The Actor Script will be as simple as this:

import asyncio
import time
import logging
from apify.log import ActorLogFormatter
from src.main import main, Actor

handler = logging.StreamHandler()
handler.setFormatter(ActorLogFormatter())

apify_logger = logging.getLogger('apify')
apify_logger.setLevel(logging.DEBUG)
apify_logger.addHandler(handler)

asyncio.run(main())

The logger is defined using the Apify instance instead, and no inputs are used in the main function because they will be defined inside it, as we are going to see in the next step.

Step 2 — Create a main.py script

As we have seen previously, the main function takes three inputs, and looks like this for the Default Script:

""" This script contains the function to scrape Polymarket """
# pylint: disable=W0718
# pylint: disable=W0621
import json
from playwright.async_api import async_playwright

async def main(logger, search_input, num_markets):
    """ Main function to scrape Polymarket """
    async with async_playwright() as pw:

        # Creates an instance of the Chromium browser and launches it
        browser = await pw.chromium.launch(headless=False)

        # Creates a new browser page (tab) within the browser instance
        page = await browser.new_page(locale='en-US')
        logger.info("Open Polymarket")
        if search_input == "Top":
            await page.goto("https://polymarket.com/markets/all")
            await page.wait_for_timeout(4000)
        if search_input == "New":
            await page.goto(
                "https://polymarket.com/markets/all?_s=start_date%3Adesc")
            await page.wait_for_timeout(4000)


        # Collect results in the loop
        for i in range(round(num_markets / 20)):

            # Iteration number
            logger.info("Iteration Number: %s", i)

            # Get response
            page.on(
                "response", lambda response: check_json(response, logger))

            # Scroll down
            await page.locator(
                ".c-dhzjXW.c-dhzjXW-idNOCmT-css"
            ).last.scroll_into_view_if_needed(timeout=5000)

            await page.wait_for_timeout(2000)

        # All markets
        all_markets = []

        # Read and process each row
        with open('temp.json', 'r', encoding='utf-8') as json_file:
            for line in json_file:

                # Parse each line as a JSON array
                row = json.loads(line.strip())

                # Process each dictionary in the row
                for dictionary in row:
                    all_markets.append(dictionary)

        # Get the sample for the output
        markets_sample = all_markets[:num_markets]

        if markets_sample:
            logger.info("Scraping Succeeded!")
            with open("result.json", 'w', encoding="utf-8") as f:
                json.dump(markets_sample, f, indent=4)

        await page.close()

At the end of the script, we see that the scraped information is saved in the result.json file, with the Actor Script this will be different, instead the results will be stored in the Apify database and available to explore and export in the console.

Let’s see how the script changes for the actor:

""" This script contains the functions to scrape polymarket """
# pylint: disable=W0718
# pylint: disable=W0621
import json
from apify import Actor
from playwright.async_api import async_playwright

async def main():
    """ Main function to scrape Polymarket """
    async with Actor:
        # Read the Actor input
        actor_input = await Actor.get_input() or {}
        search_input = actor_input.get("search_input")
        num_markets = actor_input.get("number_of_markets")

        async with async_playwright() as pw:

            # Creates an instance of the Chromium browser and launches it
            browser = await pw.chromium.launch(
                headless=False)

            # Creates a new browser page (tab) within the browser instance
            page = await browser.new_page(locale='en-US')
            Actor.log.info("Open Polymarket")
            if search_input == "Top":
                await page.goto("https://polymarket.com/markets/all")
                await page.wait_for_timeout(4000)
            if search_input == "New":
                await page.goto(
                    "https://polymarket.com/markets/all?_s=start_date%3Adesc")
                await page.wait_for_timeout(4000)

            # Clean JSON file
            with open("temp.json", "w", encoding="utf-8") as file:
                pass  # This leaves the file completely empty

            # Collect results in the loop
            for i in range(round((num_markets/2) / 20)):

                # Iteration number
                Actor.log.info("Iteration Number: %s", i)

                # Get response
                page.on(
                    "response", check_json)

                try:
                    # Scroll down
                    await page.locator(
                        ".c-dhzjXW.c-dhzjXW-idNOCmT-css"
                    ).last.scroll_into_view_if_needed(timeout=10000)

                except Exception as error:
                    Actor.log.error(str(error))
                    await page.wait_for_timeout(2000)
                    continue

            await page.wait_for_timeout(2000)

            await page.close()

            # All markets
            all_markets = []

            # Read and process each row
            with open('temp.json', 'r', encoding='utf-8') as json_file:
                for line in json_file:

                    # Parse each line as a JSON array
                    row = json.loads(line.strip())

                    # Process each dictionary in the row
                    for dictionary in row:
                        all_markets.append(dictionary)

            # Drop duplicates
            all_markets = [
                dict(t) for t in
                {tuple(sorted(d.items())) for d in all_markets}]

            # Get the sample for the output
            markets_sample = all_markets[:num_markets]
            Actor.log.info("Total Scraped: %s", len(all_markets))
            Actor.log.info("Sample Length: %s", len(markets_sample))

            Actor.log.info("Scraping Succeeded!")
            await Actor.push_data(markets_sample)

Not so much has changed, but the first thing we notice is that the inputs are defined within the function using the actor_input object. You might be wondering where the strings search_input and number_of_markets were defined. And they actually come from the input_schema.json that we are going to take a look next.

Another change is that all logger objects are replaced by the Actor.log. Finally, as I mentioned, the data is not stored as JSON but pushed instead using the Actor.push_data() function.

Step 3— Create an input_schema.json

When you publish your actor, a proper input schema is needed so the user can have the option to manually insert the right inputs for scraping. The input_shema.json is a file that allows you to customize the entries, and the full documentation can be seen here:

For the Polymarket actor, all I need is a dropdown menu to select the search input and a field to enter the number of markets. Therefore, my structure looks like this:

{
    "title":"Polymarket Scraper",
    "schemaVersion": 1,
    "type": "object",
    "properties": {
      "search_input": {
        "title": "Search Input (Top or New)",
        "type": "string",
        "editor": "select",
        "default": "Top",
        "enum": ["Top", "New"],
        "enumTitles": ["Top", "New"],
        "description": "Search Polymarket for 'Top' or 'New' markets."
      },
      "number_of_markets": {
        "type": "integer",
        "title": "Number of Markets",
        "description": "The number of markets (posts) to be scraped.",
        "default": 100,
        "minimum": 100,
        "maximum": 1500
      }
    },
    "required": ["search_input", "number_of_markets"]
  }

Step 4 — Create a Dockerfile

Since we are using Python, we need a Docker image with a Python version and install some dependencies. For that, you can use this default Dockerfile, which you can also find in the templates repository:

# First, specify the base Docker image.
# You can see the Docker images from Apify at https://hub.docker.com/r/apify/.
# You can also use any other image from Docker Hub.
FROM apify/actor-python-playwright:3.11

# Second, copy just requirements.txt into the Actor image,
# since it should be the only file that affects the dependency install in the next step,
# in order to speed up the build
COPY requirements.txt ./

# Install the packages specified in requirements.txt,
# Print the installed Python version, pip version
# and all installed packages with their versions for debugging
RUN echo "Python version:" \
 && python --version \
 && echo "Pip version:" \
 && pip --version \
 && echo "Installing dependencies:" \
 && pip install -r requirements.txt \
 && echo "All installed Python packages:" \
 && pip freeze

 # Install Playwright and its dependencies
RUN playwright install-deps && \
    playwright install

# Next, copy the remaining files and directories with the source code.
# Since we do this after installing the dependencies, quick build will be really fast
# for most source file changes.
COPY . ./

# Use compileall to ensure the runnability of the Actor Python code.
RUN python3 -m compileall -q .

# Specify how to launch the source code of your Actor.
# By default, the "python3 -m src" command is run
CMD ["python3", "-m", "src"]

The final step is to customize the actor.json. Let’s see what you can do!

Step 5— Customize the actor.json and deploy the actor

Previously, when we did apify init an actor.json file was created, with a default title, version, and so on, but now we need to specify the right project’s title, the path for the Dockerfile, and the path to the input schema. In addition, you may want to have a structured output for the user, and that is also configured in this file. For that, you need to label and give a format to all the output variables. Let’s see an example of actor.json:

{
    "actorSpecification": 1,
    "name": "polymarket-scraper",
    "version": "0.1",
    "buildTag": "latest",
    "input": "./input_schema.json",
    "dockerfile": "./Dockerfile",
    "storages": {
        "dataset": {
            "actorSpecification": 1,
            "views": {
                "overview": {
                    "title": "Overview",
                    "transformation": {
                        "fields": [
                            "market_question",
                            "market_creation_date",
                            "market_end_date",
                            "market_link",
                            "market_description",
                            "market_chance",
                            "market_liquidity",
                            "market_volume",
                            "market_volume_24h",
                            "market_one_day_price_change",
                            "market_yes_price",
                            "market_no_price",
                            "market_spread",
                            "market_closed",
                            "market_rewards_amount",
                            "market_rewards_daily_rate",
                            "market_num_comments"
                        ]
                    },
                    "display": {
                        "component": "table",
                        "properties": {
                            "market_question": {
                                "label": "Market Question",
                                "format": "text"
                            },
                            "market_creation_date": {
                                "label": "Market Creation Date",
                                "format": "date"
                            },                            
                            "market_end_date": {
                                "label": "Market End Date",
                                "format": "date"
                            },  
                            "market_link": {
                                "label": "Market Link",
                                "format": "link"
                            },
                            "market_description": {
                                "label": "Market Description",
                                "format": "text"
                            },
                            "market_chance": {
                                "label": "Market Chance",
                                "format": "text"
                            },
                            "market_liquidity": {
                                "label": "Market Liquidity",
                                "format": "number"
                            },
                            "market_volume": {
                                "label": "Market Volume",
                                "format": "number"
                            },
                            "market_volume_24h": {
                                "label": "Market Volume 24h",
                                "format": "number"
                            },
                            "market_one_day_price_change": {
                                "label": "Market One Day Price Change",
                                "format": "number"
                            },
                            "market_yes_price": {
                                "label": "Market Yes Price",
                                "format": "number"
                            },
                            "market_no_price": {
                                "label": "Market No Price",
                                "format": "number"
                            },                            
                            "market_spread": {
                                "label": "Market Spread",
                                "format": "number"
                            },
                            "market_closed": {
                                "label": "Market Closed",
                                "format": "text"
                            },
                            "market_rewards_amount": {
                                "label": "Market Rewards Amount",
                                "format": "number"
                            },
                            "market_rewards_daily_rate": {
                                "label": "Market Rewards Daily Rate",
                                "format": "number"
                            },
                            "market_num_comments": {
                                "label": "Market Number of Comments",
                                "format": "number"
                            }
                        }
                    }
                }
            }
        }
    }
}

This is the final step to build your actor, and you can test it locally with the following CLI command:

apify run --purge --input '{"search_input": "Top", "number_of_markets": 100}'

Your output will be stored in the storage folder previously created. If everything works smoothly, you can finally deploy to the Apify cloud instance:

apify push

Sometimes you may see some errors in the console that you didn't get locally. It may be related to your input schema, thus make sure the file is properly configured with this command:

 apify validate-schema 

Publish and test the actor on Apify

Now, everything should be properly configured in the repository. But in order to publish your actor, you must add display information and a monetization plan like below:

Once your actor strength is excellent, you can test it for free in the Apify console.

With the output structure I’ve defined in the actor.json file, I get the following table:

The actor will pass through some quality checks, and if any significant issues are found, you’ll receive a notification indicating that the actor is under maintenance and requires updates. Otherwise, you’re all set, and your web scraping script is ready to be monetized and made available to users worldwide!

Need help with automation, AI, or data science? Let’s talk.

Book a 20-min consultation

Conclusion

After following this tutorial, you realise that creating an Apify actor isn’t too complicated. You can preserve most of your default web scraping script, with only some parts that need to be replaced by the SDK.

Nonetheless, it is worth paying special attention to the actor.json structure and the input_schematic.json so that your deployment doesn’t cause any issues. Also, make sure that your README.md file clearly explains your web scraping inputs and outputs.

With that being said, there’s no need to leave your scripts gathering mold, you can just deploy them on Apify, and maybe someone, somewhere, will make good use of them.

The blue logo and the application’s interface make BlueSky a twin brother of Twitter (currently X), with slight changes. Users who are familiar with X can easily jump in and start using it without spending time understanding how it works.

The platform has been experiencing significant growth since the U.S. elections in 2024, which is probably due to Elon Musk actively supporting the republican party. Some on the other side of the coin have decided to abandon the ship or look for alternatives.

As of January 2025, BlueSky reported having over 27 million registered users, with 55.7% being U.S. citizens.

Aside from people moving due to politics, BlueSky comes with features and tools that are certainly important to take a look at:

• Custom feeds: In BlueSky, users can create custom feeds to a have more personalized experience and choose what they want to see. This also minimizes the algorithm's effort trying to predict your preferred topics.

• More control: Users have more control over what they can see, mute certain words or tags, and hide unpleasant comments in their posts. These and other features that offer more power to the user are a big plus when compared to X.

• Decentralization: The platform is built on the Authenticated Transfer (AT) Protocol, which enables users to host their data on servers of their choice, facilitating the creation of clients, bots, and integrations.

• Free of Advertisements: On BlueSky, you don’t need to be bombarded with ads every time.

• Free and simple API: As a developer, this one is my favorite. Unlike X, which requires users to pay for their API access and you need to wait for validation, BlueSky makes the process outstandingly easy. You only need your login username and password to connect to the API, currently available in Python and TypeScript.

I didn’t know much about BlueSky until recently, but I was contacted through Upwork by someone who wanted to create a bot on this platform to show daily updates about the egg prices in the United States.

I immediately accepted the job proposal and started working with the Python SDK to make automated posts with a seaborn chart showing the egg prices over time.

Before heading to the BlueSky API, the workflow starts with some web scraping. Let’s see what I’ve done to get updated information, and what tools I’ve used.

Scrape the daily egg price from reliable sources

Obtaining the data for this project can take several distinct paths. Some may be more time-consuming than others, but could provide more accuracy. For instance, we could scrape content from several supermarkets and grocery shops across all states of the U.S. and then obtain the mean value.

I took a simpler path, not because I did not like the first idea, but because the main goal was to deliver an actionable bot and not be too picky with the data accuracy.

Trading Economics is a well-known website for observing trends in commodities and stocks. The company provides an API and Python SDK, which allows you to get historical data across different markets and is worth paying for more advanced use cases. In this project, I only needed the daily price, which I could grab from the main page.

Do you see the price in blue over the chart? That’s the exact value I needed to scrape, and for that, I used Playwright headless browser with its built-in HTML parser capabilities.

First, we need to download the package using the following command line:

pip install playwright

Next, we need to install the browser using this command:

playwright install

It usually asks for some dependencies to be installed as well:

playwright install-deps

Now let’s see the Python script used to get the price value:

import json
from playwright.async_api import async_playwright
from datetime import datetime
import pytz

# Add list of user agents
USER_AGENTS = [
    ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
     "(KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"),
    ("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
     "(KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"),]

async def run_scraper():

    # Get current US date
    us_tz = pytz.timezone('America/New_York')
    current_date = datetime.now(us_tz).strftime('%Y-%m-%d')
    
    async with async_playwright() as pw:
        browser = await pw.chromium.launch(
            headless=True,  # Run headless in production
            args=['--no-sandbox', '--disable-setuid-sandbox']
        )

        # Creates a new browser page with random user agent
        context = await browser.new_context(
            user_agent=USER_AGENTS[0],
            viewport={'width': 1920, 'height': 1080},
            locale='en-US'
        )
        page = await context.new_page()
        await page.goto("https://tradingeconomics.com/commodity/eggs-us")
        egg_price = await page.locator(
            '.iChart-bodylabels-ohlc').first.inner_text()
        egg_price = f"${egg_price.split(' ')[-1].strip()}"
        
        # Push data to JSON file
        data_entry = {
            "date": current_date,
            "egg_price": egg_price
        }
        
        try:
            with open('data.json', 'r', encoding='utf-8') as f:
                data = json.load(f)
        except FileNotFoundError:
            data = []
        
        data.append(data_entry)

        with open('data.json', 'w', encoding='utf-8') as f:
            json.dump(data, f, indent=4)
        
        await browser.close()

The reason why I often use USER_AGENTS is because they help avoid Cloudfare detections and other types of anti-bot blocking solutions.

Inside the run_scraper() function, the most important is the way I grabbed the value using the Playwright's page instance. First, I had to find the HTML class representing the daily price value.

As you can see in the image above the class I needed is iChart-bodylabels-ohlc , therefore, I used the .locator() function to get that element and the .inner_text() function to get the text inside it. Finally, I cleaned the output to obtain the price value only.

Another key step in the script is the United States timezone retrieval. For that, I used the package pytz which you can also install via pip. This, paired with the datetime library provides the exact day and time in the New York timezone. The rest of the code shows how the price and the date are saved into a JSON file every time the code runs.

The scraping function is accessed from the main script that contains the BlueSky API handling. Let’s see how it was built in the next chapter.

Create the chart and main Python script

The BlueSky post is made of three main parts — the post text, an image/chart, and the hashtags. When the main script is triggered, the output looks like this:

The sequence of the script is the following:

1. A logging function is created and launched to track and monitor the bot’s activity.

2. A login function, try_login(), is called with a certain number of retries because the API doesn’t always work in the first run.

3. The run_scraper() function is triggered to scrape the daily value and add it to the JSON file.

4. A chart is created and saved as PNG, using seaborn and matplotlib libraries.

5. Dates and price values are preprocessed with several conditions that will return variables to use in the post text.

6. Byte positions are created for the hashtags.

7. Finally, the function try_post() is triggered, and the post is submitted to BlueSky.

Before having a better grasp of the above steps, we first need to install the BlueSky Python SDK:

pip install atproto

The functions that use the BlueSky API are try_login() and try_post(), and they will help us understand the remaining steps.

import os
from atproto import Client
from dotenv import load_dotenv

def try_login(max_retries=3, delay=60):
    """Try to login with retries"""
    logger.info("Starting login process...")
    for attempt in range(max_retries):
        try:
            client = Client()
            client.login(os.getenv('username'), os.getenv('password'))
            logger.info(f"Successfully logged in on attempt {attempt + 1}")
            return client
        except Exception as e:
            if attempt < max_retries - 1:
                logger.warning(f"Login attempt {attempt + 1} failed: {str(e)}")
                logger.info(f"Retrying in {delay} seconds...")
                time.sleep(delay)
            else:
                logger.error(f"Failed to login after {max_retries} attempts: {str(e)}")
                raise Exception(f"Failed to login after {max_retries} attempts: {str(e)}")

def try_post(client, image_data, text, facets, max_retries=3, delay=60):
    """Try to post with retries"""
    logger.info("Starting post attempt...")
    for attempt in range(max_retries):
        try:
            post = client.send_image(
                text=text,
                facets=facets,
                image=image_data,
                image_alt="US Egg Prices",
                langs=['en']
            )
            logger.info(f"Successfully posted on attempt {attempt + 1}")
            return post
        except Exception as e:
            if attempt < max_retries - 1:
                logger.warning(f"Post attempt {attempt + 1} failed: {str(e)}")
                logger.info(f"Retrying in {delay} seconds...")
                time.sleep(delay)
            else:
                logger.error(f"Failed to post after {max_retries} attempts: {str(e)}")
                raise Exception(f"Failed to post after {max_retries} attempts: {str(e)}")

We see above that the try_login() function, not only makes the API connection with our BlueSky credentials, it also returns the client object, which is used to access other functions like .send_post() or .send_image() in this case.

The try_post() function, takes the image previously saved, the client object, text, and faucets — highlight hashtags in the post — as inputs, and submits the posts on BlueSky.

At this point, adding the image as input requires you to read it as a binary file:

# Read the image data from the file
with open('egg_prices_chart.png', 'rb') as f:
    image_data = f.read()

The complexity of the text input depends on the amount of information you wish to include. As for the faucets, which is probably the most confusing term, is a list of dictionaries that specify the positions of mentions, hashtags, and links in the text. Otherwise, they are submitted as pure strings. This list can be created in the following way:

# Find all hashtags and their positions
hashtags = ["#eggs", "#priceofeggs", "#H5N1", "#economics", "#usa"]
facets = []

# Calculate byte positions for each hashtag
for hashtag in hashtags:
    start_pos = post_text.find(hashtag)
    if start_pos != -1:
        facets.append({
            "index": {
                "byteStart": start_pos,
                "byteEnd": start_pos + len(hashtag)
            },
            "features": [{
                "$type": "app.bsky.richtext.facet#tag",
                "tag": hashtag[1:]  # Remove the # symbol
            }]
        })

Knowing these steps and functions should be enough to start posting on BlueSky from your machine, but you can perform many more actions. Just explore their insightful API documentation.

Let’s not forget one final and important note. If the goal is to create a bot it must be automated to run on a set schedule — hourly, daily, weekly, or as needed. There are multiple ways to achieve this, such as using Replit, an online IDE and deployment platform, to schedule execution. In my case, I run it on my Contabo VPS by configuring a cron job that triggers daily at a specified time.

Need help with automation, AI, or data science? Let’s talk.

Book a 20-min consultation

Conclusion

As the number of users keep growing on BlueSky and other social media platforms are already saturated, perhaps creating an account and starting using their API could be a clever way to get engagement fast and build a strong community.

For example, if you own a website about medical devices, developing a bot to share blog posts and daily updates could drive more traffic and generate valuable leads.

You can also create AI agents that post and respond intelligently based on semantic context. The possibilities are limitless, especially in this era of RAG systems and AI agents.

As a result, crypto articles can become outdated quickly. What’s relevant one week may not be the next. Therefore, writers must continuously monitor trends and produce fresh content at a rate that is hard to see in other technological fields.

This constant stream of cryptocurrency, blockchain, and web3 news serves as a valuable resource for those looking to inform their investment strategies, develop apps around trending topics, analyze market sentiment, or simply stay up-to-date on the latest ecosystem advancements.

We can get the latest news with web scrapers, but what about fetching the information that we’re looking for without spending time reading through several articles or making extensive Google searches?

To address this challenge, I created a Retrieval-Augmented Generation (RAG) system specifically designed for crypto news. This system enables users to query topics related to cryptocurrencies, blockchain, and web3, and receive the most up-to-date information on those subjects.

For example, if you ask ChatGPT to explain Decentralized Finance (DeFi), it can provide a detailed and comprehensive answer. However, if you ask about the latest Ethereum price predictions, the chat interface may need to search the web, and its response will likely be limited.

In this piece, I will guide you through the steps I took to build CryptoRAG, a chatbot tailored to get the latest information about cryptocurrencies, blockchain and web3.

We’ll start by exploring the fundamentals of RAG systems and the specific architecture I implemented. Next, we’ll cover how to upload the data to the Chroma database by creating Langchain documents. In addition, we’ll look at the query function, the prompt template, and two different ways of using Llama3.3 in the pipeline. Finally, we’ll bring everything together by creating a user interface to enable question-answering capabilities.

Hands-on with my CryptoRAG architecture

Large Language Models (LLMs) are trained on huge volumes of publicly available data on the internet, and use countless parameters to originate outputs that can answer your questions, create original content, or simply communicate with you. However, just like human beings, these models don’t have all the information in the world, and certainly not the latest one.

Retrieval-Augmented Generation (RAG) fixes this by integrating external data from a specific domain, document, or other sources with a Large Language Model (LLM). This hybrid approach enables the model to produce outputs that are tailored to a particular field or industry.

The RAG process is split into 3 main modules

1. Retrieval Module: This module fetches and ranks relevant data within a vector database based on a given input. During this process, the input query is analyzed, and the system searches through indexed data to retrieve information that aligns with the query, prioritizing relevance.

2. Augmentation Module: This second module plays a crucial role in enhancing the performance of the RAG system. Its function is to take the top-ranked data retrieved from the Retrieval Module and seamlessly integrate it into the prompt that feeds the Large Language Model (LLM). By adding the relevant external data to the prompt, the Augmentation Module provides the LLM with additional context and information, enabling more accurate, informative, and relevant responses.

3. Generation Module: This module generates a response by combining its language capabilities with newly retrieved external data, which is incorporated in the prompt template for the LLM. The result is a response based on the external data but cleaned and explained like the LLM would do for it’s default trained data. Depending on the prompt template, the output can show sources and other metadata features.

Based on the above steps, my CryptoRAG architecture ended up like this:

Outside the RAG modules, there’s the Web Scraping script I run daily to get the latest cryptocurrency news using my Cointelegraph Scraper on Apify.

Since I don’t need a big storage solution, a simple Google Sheet serves to retain the crypto news data temporarily.

Then I used a script to extract the data from the Google Sheet and start the encoding process, where I used Langchain to split the data into chunks and add metadata. These chunks are then vectorized using a well-known embedding model from Hugging Face: sentence-transformers/all-MiniLM-L6-v2. At this point, we are already in the Retrieval Module.

Once the chunks are vectorized, they are stored in the Chroma vector database. I chose Chroma because it is a completely open-source solution with no associated costs, making it an accessible and budget-friendly option.

When an input is provided to the RAG system, it is also encoded using the same embedding model, and the most relevant chunks of information are fetched from the Chroma database using similarity search. In other words, it compares vectors of information, and the closest numbers match the information that is most related to the input.

Now we enter the Augmentation Module, where the most similar documents are used to create a prompt template to feed into the LLM model (Llama 3.3). Later, we’ll explore how to design an effective prompt and use the model, whether for free or via paid APIs.

Once the prompt template is applied to the LLM, it generates a response to the initial query, completing the Q&A process. The input and outputs are connected through a simple chat interface with some suggestions for those starting to use the app.

Access the data and create documents with Langchain

As mentioned previously, the scraper uploads the data to a Google Sheet. To access it, some configurations in the Google Cloud Platform must be done beforehand, such as creating a service account and generating credentials. You can learn more about it here.

Once done, we can use gspread Python library to access the Google Sheet and extract the information as a Pandas dataframe.

import pandas as pd
import gspread
from google.oauth2.service_account import Credentials

class GoogleAccess:
    """ Class containing the Google interaction functions """

    @staticmethod
    def google_authentication(credentials) -> object:
        """ This function gives authentication to the Google Account.

        Scopes defines the permissions to Google Sheets and Google Drive.
        Than, using gspread, it authorizes the authentication and
        opens the worksheet to work with.

        """
        # Authenticate with Google Sheets using the JSON key file
        scope = ['https://www.googleapis.com/auth/spreadsheets',
                'https://www.googleapis.com/auth/drive']

        creds = Credentials.from_service_account_file(
            credentials, scopes=scope)

        client = gspread.authorize(creds)

        return client

    @staticmethod
    def read_from_sheet(client, sheet_id) -> pd.DataFrame:
        """ This function reads the data from the Google Sheet. """

        # Open the Google Sheet by id
        sheet = client.open_by_key(sheet_id)

        # Access the worksheet name
        ws = sheet.worksheet('main')

        # Create a dataframe
        df = pd.DataFrame(data=ws.get_all_records())
        return df

The credentials are stored in a JSON file that contains the service account information. To create Langchain documents, we just need to use the credentials file along with the Google Sheet ID as inputs in this function:

from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.schema import Document

def create_documents(credentials_google, google_sheet_id):
    """ This function splits the articles in chunks 
    from a dataset obtained by reading a google sheet. 
    And creates documents with each chunk.
    """

    # Create client
    client = GoogleAccess.google_authentication(credentials_google)
    df = GoogleAccess.read_from_sheet(client, google_sheet_id)
    df['id'] = df['id'].astype('str')

    # Initialize the text splitter
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=5000,
        chunk_overlap=0)

    # Create a list of LangChain documents
    documents = []

    for _, row in df.iterrows():
        content_chunks = splitter.split_text(row['content'])
        for i, chunk in enumerate(content_chunks):
            documents.append(Document(
                page_content=chunk,
                metadata={
                    "category": row['category'],
                    "link": row['link']
                },
                id=f"{row['id']}{i}"
            ))

    return documents

For each article (row in the dataframe), I divide it into chunks of 5,000 characters. I opted for larger chunks because smaller ones seemed to negatively impact my RAG’s performance, resulting in poor-quality information about the articles. I also included the link in the metadata to embed the sources of information directly into the final chatbot interface.

Upload the encoded documents to the Chroma vector database

The previous functions allowed me to create the Langchain documents, but I still needed to vectorize and upload them to the vector database. This is achieved in the following script:

import os
from langchain_huggingface.embeddings import HuggingFaceEmbeddings
from src.preprocess import create_documents
from langchain_chroma import Chroma

load_dotenv('keys.env')

def update_chromadb():
    """Update ChromaDB with new documents by deleting existing
      collection and creating a new one."""
    try:
        # Create temporary directory for new database
        temp_dir = "chroma_temp"
        os.makedirs(temp_dir, exist_ok=True)

        # Initialize Hugging Face embeddings
        embedding_model = HuggingFaceEmbeddings(
            model_name="sentence-transformers/all-MiniLM-L6-v2")

        # Create new documents
        documents = create_documents(
            'credentials.json', os.getenv("SHEET_ID"))

        # Create new database in temporary location
        Chroma.from_documents(
            documents=documents,
            collection_name="cointelegraph", 
            embedding=embedding_model,
            persist_directory=temp_dir
        )

        # If successful, replace old database with new one
        if os.path.exists("chroma"):
            shutil.rmtree("chroma")
        shutil.move(temp_dir, "chroma")

        logger.info(
            "Successfully refreshed ChromaDB collection with new documents")
        
        return True
        
    except Exception as e:
        logger.error("Error updating ChromaDB: %s", str(e))
        if os.path.exists(temp_dir):
            shutil.rmtree(temp_dir)
        raise

The chroma object, integrated with Langchain, is instantiated using the .from_documents() function to upload the previously created documents and simultaneously embed them using the sentence-transformers/all-MiniLM-L6-v2 model from Hugging Face.

The data is stored in the cointelegraph collection, which works like a table in SQL databases. During the process, the data is temporarily stored in a folder named chroma_temp. If the documents are successfully updated, they replace the existing data in the main database folder, chroma.

At the moment of launching the application, the following function is triggered to get the Chroma db object without making any changes in the database:

def initialize_chatbot():
    """Initialize ChromaDB and embeddings."""
    global db, embedding_model, last_update

    try:
        # Initialize Hugging Face embeddings
        embedding_model = HuggingFaceEmbeddings(
            model_name="sentence-transformers/all-MiniLM-L6-v2")

        # Load existing database
        db = Chroma(
            collection_name="cointelegraph",
            embedding_function=embedding_model,
            persist_directory="chroma"
        )
        
        if not last_update:
            last_update = datetime.now()
            
        logger.info("Successfully loaded ChromaDB collection")
    except Exception as e:
        logger.error("Error initializing ChromaDB: %s", str(e))
        raise

This db object will be used to query the vector database and obtain a response, which we will see in the next chapter.

Query the RAG system and get a detailed output

This is the part where we finally use the LLM, and for that, I tried two approaches. The first one is free to use with Ollama. But if you lack a GPU, chances are that the model will take too much computational time. The second approach is using OpenRouter models and the OpenAI Python library.

Use an LLM for free in your RAG system

Let’s start by exploring the free method using Ollama. You first need to install it via curl in your WSL or Linux machine with this command:

curl -fsSL https://ollama.com/install.sh | sh

Next, run the following command to start Ollama:

ollama serve

Fetch the Ollama model you want to use:

ollama run llama3.3

The model Llama3.3 is huge! So either you have enough storage (and a GPU) to run it, or I recommend you look for lightweight Ollama model alternatives, such as mistral.

Now guess what? There’s also a Langchain python package for Ollama, so let’s install it:

pip install langchain-ollama

With this configuration, our query function would look like this:

from langchain_core.prompts import PromptTemplate
import os
from langchain_ollama.llms import OllamaLLM


def query_rag(query_text, db, prompt, model_name):
    """
    Query a Retrieval-Augmented Generation (RAG) system using Chroma database,
    and format the response with links and categories.
    """

    # Retrieving the context from the DB using similarity search
    results = db.similarity_search_with_relevance_scores(query_text, k=3)

    # Check if there are any matching results or if
    # the relevance score is too low
    if len(results) == 0 or results[0][1] < 0.2:
        return (
            "No relevant information found. Try to ask something "
            "related to crypto, blockchain and web3.")

    else:
        # Combine context from matching documents
        context_text = "\n\n - -\n\n".join(
            [doc.page_content for doc, _score in results]
        )

        # Extract categories and links from metadata
        categories = ", ".join(
            {doc.metadata.get("category", "Unknown") for doc, _ in results})
        links = ", ".join(
            {doc.metadata.get("link", "N/A") for doc, _ in results})

        # Create and format the prompt
        template = PromptTemplate(
            input_variables=["context", "question", "categories", "links"],
            template=prompt,
        )
        prompt = template.format(
            context=context_text,
            question=query_text,
            categories=categories,
            links=links
        )

        # Use the language model to generate a response
        model = OllamaLLM(
           model=model_name, cache=False, verbose=True, num_ctx=500)
        response_text = model.invoke(prompt)

        # Format and return response including generated text and sources
        formatted_response = f"{response_text}\nSources: {links}"

        return formatted_response

Note that the model_name input you see in the script above, should be a string with the name of the model you are running in Ollama, for instance llama3.3.

Use OpenRouter models with OpenAI

If you don’t have a GPU, like me, the best option is to use a paid service. I prefer OpenRouter because it allows me to purchase credits with crypto and works seamlessly with the OpenAI Python package:

pip install openai

Once the package is installed, you need to generate a token in your OpenRouter account and change the query function to this instead:

from langchain_core.prompts import PromptTemplate
import os
from openai import OpenAI
from dotenv import load_dotenv

load_dotenv("keys.env")


def query_rag(query_text, db, prompt):
    """
    Query a Retrieval-Augmented Generation (RAG) system using Chroma database,
    and format the response with links and categories.
    """

    # Retrieving the context from the DB using similarity search
    results = db.similarity_search_with_relevance_scores(query_text, k=3)

    # Check if there are any matching results or if
    # the relevance score is too low
    if len(results) == 0 or results[0][1] < 0.2:
        return (
            "No relevant information found. Try to ask something "
            "related to crypto, blockchain and web3.")

    else:
        # Combine context from matching documents
        context_text = "\n\n - -\n\n".join(
            [doc.page_content for doc, _score in results]
        )

        # Extract categories and links from metadata
        categories = ", ".join(
            {doc.metadata.get("category", "Unknown") for doc, _ in results})
        links = ", ".join(
            {doc.metadata.get("link", "N/A") for doc, _ in results})

        # Create and format the prompt
        template = PromptTemplate(
            input_variables=["context", "question", "categories", "links"],
            template=prompt,
        )
        prompt = template.format(
            context=context_text,
            question=query_text,
            categories=categories,
            links=links
        )

        client = OpenAI(
            base_url="https://openrouter.ai/api/v1",
            api_key=os.getenv("API_KEY"),
            )

        completion = client.chat.completions.create(
            model="meta-llama/llama-3.3-70b-instruct",
            messages=[
                {
                "role": "user",
                "content": prompt
                }
            ]
        )

        response_text = completion.choices[0].message.content

        formatted_response = f"{response_text}\nSources: {links}"

        return formatted_response

This approach produces much faster results, than the free one, but also comes with a cost. For instance, using 2,495 tokens in the prompt, I paid $0.000433.

Build the right prompt template

As you have seen in the previous scripts, the query_rag() function takes a prompt as input. There are many ways of building it, mine took this direction:

PROMPT_TEMPLATE_1 = """
    You are an expert assistant. Using the provided context from the 
    database, answer the question in detail. Write your response without 
    referencing articles, documents, or sources explicitly. Never start or 
    mention the following: "In the piece", "In the article" and so on in the 
    response. Provide clear and concise answers that are helpful and
    relevant to the question.

    Context:
    {context}

    Question:
    {question}

    Answer:
    - Detailed response: 
    - Relevant categories: {categories}
    - Links to explore further: {links}
    """

I had to explicitly state that I didn’t want the answer to begin by mentioning articles, as that happened a few times. Additionally, I wanted to integrate the context, the question, and the metadata information in the final output.

To create a template prompt for the model, I had to install langchain_core:

pip install langchain-core

Once done, I use the PromptTemplate instance to create the final prompt object that you can see in the previous scripts (both paid and free versions).

Build the chat interface with Flask and Cursor IDE

I’ve decided to use Python for the backend because it eases the process of connecting to Langchain, LLM packages and Chroma database. Therefore, I used the Flask framework to run the server.

In addition, I used the capabilities of Cursor composer to help me build a robust app.py script that runs daily, because I need the latest information to be uploaded to the Chroma database while the application runs.

The final main function looks like this:

""""Flask web application for Web3 & Cryptocurrency Expert Chatbot."""
# pylint: disable=W0718
import logging
import sys
import os
from langchain_chroma import Chroma
from langchain_huggingface.embeddings import HuggingFaceEmbeddings
from flask import Flask, request, jsonify, render_template, send_from_directory
from flask_cors import CORS
from src.pipeline import query_rag
from src.prompts import PROMPT_TEMPLATE_1
from flask_apscheduler import APScheduler
from datetime import datetime
from src.db_handler import update_chromadb

class Config:
    SCHEDULER_API_ENABLED = True

scheduler = APScheduler()

# Configure logging with more detailed format
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
    handlers=[
        logging.StreamHandler(sys.stdout),
        logging.FileHandler('app.log')
    ]
)
logger = logging.getLogger(__name__)

# Initialize Flask app
app = Flask(__name__, static_folder='static', static_url_path='/static')
app.config.from_object(Config())
CORS(app)

# Initialize global variables for database and embedding model
db = None
embedding_model = None
last_update = None

def update_db_task():
    """Background task to update the database"""
    global db, last_update
    try:
        logger.info("Starting scheduled database update...")
        
        # Update the database
        update_chromadb()
        
        # Reinitialize the database connection
        initialize_chatbot()
        
        last_update = datetime.now()
        logger.info("Scheduled database update completed successfully")
    except Exception as e:
        logger.error(f"Error in scheduled database update: {str(e)}")

@app.route('/')
def home():
    """Render the chat interface."""
    try:
        return render_template('index.html')
    except Exception as e:
        logger.error("Error serving home page: %s", str(e))
        return "Internal server error", 500

@app.route('/chat', methods=['POST'])
def chat():
    """Handle chat messages using RAG system."""
    try:
        if db is None or embedding_model is None:
            logger.error("Chatbot components not properly initialized")
            return jsonify({'error': 'Chatbot not initialized properly'}), 503

        data = request.get_json()
        if not data or 'message' not in data:
            logger.warning("Invalid request: missing message")
            return jsonify({'error': 'No message provided'}), 400

        user_message = data['message'].strip()
        logger.info("Processing chat message: %s...", user_message[:50])

        # Use RAG system to generate response
        formatted_response= query_rag(
            user_message, db, PROMPT_TEMPLATE_1)

        logger.info("Successfully generated response")
        return jsonify({
            'response': formatted_response,
        })

    except Exception as e:
        logger.error("Error processing chat message: %s", str(e))
        return jsonify(
            {'error': 'An error occurred while processing your request'}), 500


if __name__ == '__main__':
    try:
        # Clear log files at startup
        clear_log_files()
        
        # Initialize scheduler
        scheduler.init_app(app)
        
        # Schedule database updates every 10 minutes
        scheduler.add_job(
            id='update_db',
            func=update_db_task,
            trigger='interval',
            days=1,
            next_run_time=datetime.now()  # Run once immediately
        )
        
        scheduler.start()

        # Initialize chatbot components
        logger.info("Starting application initialization...")
        initialize_chatbot()        
        
        logger.info(
            "Chatbot initialization and "
            "scheduler setup completed successfully")

        # Start Flask server
        logger.info("Starting Flask server on port 5000...")
        app.run(host='0.0.0.0', port=5000, debug=False, use_reloader=False)
    except Exception as e:
        logger.error("Critical server error: %s", str(e))
        sys.exit(1)

For the front-end, I mostly relied on Cursor composer to build the interface, and then I adjusted the HTML elements myself. However, it’s clear that you don’t need to be a front-end developer to create a ready-to-go application nowadays. Here’s how the app looks like:

For deployment, I’m currently using Replit. I simply connected my GitHub account, made a few configurations, and clicked the deploy button.

Need help with automation, AI, or data science? Let’s talk.

Book a 20-min consultation

Conclusion

Building a RAG system doesn’t take much time today with the help of AI and the right tools. With so many frameworks and options available, it can be overwhelming to know where to start.

That’s why, even though it’s not the most user-friendly, I prefer sticking with Langchain. I need to install several libraries during the process, but at least I know they’re compatible and won’t break the pipeline. For vector databases, free and open-source solutions like Chroma DB work well, and when it comes to LLMs, just choose the one that fits your needs.

Concerning programming, there’s no going back, you either use Copilot, Cursor, or another AI-powered IDE, or your productivity will lag behind both developers and non-developers. It’s amazing to see how quickly we can build applications now!

I hope this tutorial helps you get started with your own RAG system and saves you time searching for the right tools. And remember, try CryptoRAG and leave your honest feedback/support if you find the chatbot useful!

I use the APIs with Python to overlay other LoRa models and produce less realistic results, like the one you see below, which is me riding a giant garlic in a green field with a watercolour painting touch.

At some point, I realised that these images could make a great book for my daughter, so I decided to take around 20 photos of her favourite toy, called Pupilo, and fine-tune a model on Replicate. Next, I generated images of Pupilo on a space exploration mission, and I manually put the images and the story together. Finally, I used Amazon KDP to publish the book, and I shipped a sample to my home address. The cover looked like this:

When the book finally arrived home, I was super excited to show it to my daughter. She saw it with her giant eyes and smiled. She crawled towards me and grabbed the book. It wasn’t long before she decided to put it in her mouth and crumple it completely!

It’s ok, it was my fault, I picked the worst quality paper, and the cover was very soft. But I didn’t give up there, I was ready to make a stronger and better book to pass my daughter’s quality checks. Not only that, I wanted to make the process completely or at least partially automated.

For that, I needed to build the following workflow:

1. Use Replicate to fine-tune a model based on a ZIP file containing 20 good-quality photos.

2. Use LLMs with Python to produce a simple and amusing story based on key inputs. Split the story into 13 paragraphs with an image prompt for each one of them.

3. Use the Replicate API with Python to generate several images for each one of the previously created prompts.

4. Manually select the images that go with each one of the paragraphs.

5. Create a Python script that takes each one of the selected images and the corresponding paragraph and generates a PDF file with the right dimensions for shipping.

6. Finally, ship the book to my address using Lulu.

In this piece, I will guide you through the process I’ve designed to make AI-generated books while showing you how to integrate the Replicate API with Python as well as how to use the hugchat API with Lamma 3.3 to generate paragraphs.

Fine-tune the FLUX.1 model on Replicate

Several platforms, such as fal.ai and mystic, offer fine-tuning for FLUX.1 and other AI models. However, I became quite familiar with Replicate, which costs around $1 per model.

The FLUX.1 model can be used for free on Hugging Face, but for fine-tuning and accessing the professional version of the model, you need to create an account on Replicate and add a billing account.

Once done, you need to collect between 14 and 20 photos of the person, animal or object you want to train on. The better the quality the better your model will be.

Once done, you need to collect between 14 and 20 photos of the person, animal or object you want to train on. The better the quality the better your model will be.

Once you have the photos, you need to compress them into a ZIP file and head to the ostris/flux-dev-lora-trainer page on Replicate.

Here you’ll need to choose a trigger word for your model to be called, and be sure to use something random like “MRCFTMGA”. For instance, if you’re called Mario and use the trigger word “MARIO”, chances are that you’ll be generating images of you driving karts and dressed in red, or worse, images of Mario and Luigi instead of you.

You can leave the default settings in most of the fields, just upload the zip file with all your photos, and run the API. Wait around 10 minutes, and you’ll have your model ready to start generating images like this

If you want to use your fine-tuned model with Python, you can click on it and you’ll see a tab for Python with all the steps to call the API.

First, you need to install the Replicate package via pip:

pip install replicate

Second, you need to export your token:

export REPLICATE_API_TOKEN=<my_token>

And finally, run the model using:

output = replicate.run(
    "macrodrigues/marco_pupilo_model:<some_code>",
    input={
        "model": "dev",
        "go_fast": False,
        "lora_scale": 1,
        "megapixels": "1",
        "num_outputs": 1,
        "aspect_ratio": "1:1",
        "output_format": "webp",
        "guidance_scale": 3,
        "output_quality": 80,
        "prompt_strength": 0.8,
        "extra_lora_scale": 1,
        "num_inference_steps": 28
    }
)
print(output)

The input settings you see above are the same settings the platform supports, so you can first take a look a them manually and see what you can adjust in the Python dictionary.

Use the HuggingChat API to build a story and generate image prompts

When it comes to using LLMs, people often think about OpenAI’s API because it is powerful and easy to implement. However, it comes with a cost. Instead, there are open-source source models that can be used completely for free with the hugchat API built by Soulter.

This API uses HuggingChat, which supports state-of-the-art models, such as Llama 3.3 from Meta and Qwen 2.5 from Alibaba Cloud.

To use it, you first need to have an account on Hugging Face. Then, you need to select the model you want to use to generate the paragraphs of the story directly in the chat’s console settings.

You can find the settings icon next to the model’s name. A window will then appear with all the available models.

Once done, we can open the IDE and create a Python function that will generate the story and image prompts:

import os
from hugchat import hugchat
from hugchat.login import Login

def gen_prompts_n_paragraphs(
        language: str,
        description: str,
        name: str):
    """ This function generate image prompts and a history. """

    # Log in to huggingface and grant authorization to huggingchat
    cookie_path_dir = "./cookies/"
    sign = Login(os.getenv('email'), os.getenv('password'))
    cookies = sign.login(cookie_dir_path=cookie_path_dir, save_cookies=True)

    # Create your ChatBot
    chatbot = hugchat.ChatBot(cookies=cookies.get_dict())

    # Input query
    input_query = (f"Write a 13-paragraph story about the character '{name}'"
                   f", based on this description:\n\n'{description}'"
                   "\n\nThe story must have a clear beginning, middle, and"
                   " end. Each paragraph should be numbered (1 to 13) and "
                   "be 18-55 words long. After each paragraph, provide a "
                   "corresponding numbered image prompt (1 to 13) that "
                   f"starts with 'A photo of {name}' and ends with "
                   "'in the style of TOK.' The image prompts must match the"
                   " content of the corresponding paragraph. Use simple, "
                   f"accessible {language} language for a story that's "
                   f"accessible for all ages and for non-{language} native"
                   " speakers, but always leave the image prompts in English. "
                   f"If you introduce new characters, always mention what the" 
                   " character/object is in the prompt.")

    res = chatbot.chat(input_query)

    # Open the file in write mode and save the text
    with open('prompt_response.txt', "w") as file:
        file.write(str(res))

The function above is one of the first in the workflow because it produces the core elements of the story. In the next chapter, we are going to see how we use the fine-tuned model with the image prompts.

Use Python with the Replicate API to generate images

The Replicate API is limited to 4 image outputs for each run. For my project, I wanted to have at least 8 images per paragraph, because the results aren’t always what we expect, so I wanted to be sure that at least one of the 8 images is decent.

Therefore, I created a script that generates 4 outputs twice in a Python loop and saves them in specific folders.

""" This fuction generates the images """
# pylint: disable=W0718
import os
import time
import requests
import replicate


def gen_images(
        img_prompts: list,
        model: str):
    """ This function generates the images with FLUX1. and 
    saves them in a folder. It takes the image prompts as inputs and
    the model. """

    retries = 0
    while retries < 5:
        try:
            break
        except Exception as error:
            print(error)
            retries += 1

    for j, prompt in enumerate(img_prompts):
        group_of_images = []
        for i in range(2):
            images = replicate.run(
                model,
                input={
                    "model": "dev",
                    "prompt": prompt,
                    "extra_lora": "lucataco/flux-watercolor",
                    "lora_scale": 1.2,
                    "num_outputs": 4,
                    "aspect_ratio": "1:1",
                    "output_format": "png",
                    "guidance_scale": 3.5,
                    "output_quality": 100,
                    "extra_lora_scale": 1,
                    "num_inference_steps": 28
                }
            )
            # Add some time to not saturate the API calls
            time.sleep(30)

            # Batch of images
            group_of_images.append(images)

        # Flatten group of images
        images_flat = [
            item for sublist in group_of_images for item in sublist]

        # Create a folder to store the batch of images
        folder_name = f"images/prompt_{j+1}"
        if not os.path.exists(folder_name):
            os.makedirs(folder_name)

        # Download and save each image
        for i, url in enumerate(images_flat):
            response = requests.get(url, timeout = 50000)
            if response.status_code == 200:

                # Define the file name and path
                file_name = f"image_{i + 1}.png"
                file_path = os.path.join(folder_name, file_name)

                # Write the image to the file
                with open(file_path, "wb") as file:
                    file.write(response.content)
                print(f"Downloaded {file_name}")
            else:
                print(f"Failed to download {url}")

        print(f"All images downloaded and saved in images/prompt_{j+1}")

You can see in the replicate dictionary that I use a second LoRa called lucataco/flux-watercolor, this creates the watercolour painting style in the images because I don’t want them to look super realistic.

This step can take up to 20 minutes and costs a few dollars to generate around 104 images. From this total, I select 15, one for each paragraph (13) and 2 for the cover.

Use the ReportLab toolkit with Python to create PDF files

With the images generated and selected, it’s time to create the book pages! Definitely, I didn’t want to manually create them with Canvas, PowerPoint or any other software, because it takes a significant amount of time.

Therefore, I created a Python script with the ReportLab toolkit to generate the body of the book. First ,I had to install the library:

pip install reportlab

Second, I put together a group of functions that correctly create the body, the title, and the dedication of the book:

import os
from PIL import Image
from reportlab.pdfgen import canvas
from reportlab.pdfbase import pdfmetrics
from reportlab.pdfbase.ttfonts import TTFont
from reportlab.pdfbase.pdfmetrics import stringWidth
from PyPDF2 import PdfMerger


def gen_book_pages(
        image_path: str,
        index: int,
        text: str):
    """ This function creates a combined pdf file with an image and the text
    in the second page. It takes the image path, an index and a paragraph."""

    logo = 'tools/book/logo_pupilo.png'
    output_image = f"outputs/image_pages/output_image_{index}.pdf"
    output_text = f"outputs/image_pages/output_text_{index}.pdf"
    custom_font = "tools/book/AbhayaLibre-Medium.ttf"

    # Desired size in pixels
    target_width, target_height = 558, 558

    # Resize the original image to 7.75 inches x 7.75 inches (558x558 pixels)
    with Image.open(image_path) as img:
        # Calculate cropping box
        width, height = img.size

        # Calculate aspect ratios
        aspect_ratio_img = width / height
        aspect_ratio_target = target_width / target_height

        # Crop the image to fit the target dimensions
        if aspect_ratio_img > aspect_ratio_target:
            # Image is wider than target, crop sides
            new_width = int(height * aspect_ratio_target)
            left = (width - new_width) // 2
            top = 0
            right = left + new_width
            bottom = height
        else:
            # Image is taller than target, crop top and bottom
            new_height = int(width / aspect_ratio_target)
            left = 0
            top = (height - new_height) // 2
            right = width
            bottom = top + new_height

        resized_img = img.crop((left, top, right, bottom))
        resized_img.save('resized_image.png', format='PNG', dpi=(500, 500))

    # Create a PDF with the resized original image
    c = canvas.Canvas(output_image, pagesize=(558, 558))
    c.drawImage('resized_image.png', 0, 0, width=558, height=558)
    c.save()

    # Register the custom TTF font
    pdfmetrics.registerFont(TTFont('CustomFont', custom_font))

    # Create a separate PDF with centered text and an additional image
    c = canvas.Canvas(output_text, pagesize=(558, 558))

    c.setFillColorRGB(0.15, 0.16, 0.16)  # Custom RGB values (between 0 and 1)    

    # Set margins and draw the text within the specified box
    left_margin = 1 * 72  # 1 inch = 72 pixels
    right_margin = 1 * 72  # 1 inch = 72 pixels
    text_width = 558 - left_margin - right_margin
    font_name = "CustomFont"
    font_size = 26

    # Split the text into lines that fit within the text box width
    words = text.split()
    lines = []
    current_line = ""
    for word in words:
        if stringWidth(
            current_line + " " + word, font_name, font_size) <= text_width:
            current_line += " " + word if current_line else word
        else:
            lines.append(current_line)
            current_line = word
    lines.append(current_line)  # Append the last line

    # Draw the text lines left-aligned within the text box
    y_position = (558 / 2) + (len(lines) * font_size) / 2  # Start from center
    for line in lines:
        c.setFont(font_name, font_size)
        c.drawString(left_margin, y_position, line)
        y_position -= font_size * 1.2

    # Add the logo
    offset_left = 0.1 * 72  # Convert inches to pixels (14.4 pixels)
    offset_bottom = 0.3 * 72  # Convert inches to pixels (21.6 pixels)
    c.drawImage(logo, offset_left, offset_bottom, 90, 90)
    c.save()

    # Combine the two PDFs
    merger = PdfMerger()
    merger.append(output_image)
    merger.append(output_text)
    merger.write(f"outputs/image_pages/output_combined_{index}.pdf")
    merger.close()

    os.remove(output_image)
    os.remove(output_text)

In the next chapter, we are going to see the final touches and how the books look on paper!

Final adjustments and shipping

All the PDFs were created respecting the dimensions of the printing platform, which in my case is Lulu. The same applies to the covers, which I decided to do manually because they need another level of detail that only a human can do.

Once I have the body and the cover, with the right dimensions, I upload both PDF files to the Lulu website, give my address, and voilá! In the next few days, the book arrives at my door.

The goal is to craft a personalized book designed specifically for an individual, which makes the printing cost higher compared to producing books for a broader audience. However, these custom books can also be ordered in bulk by schools and institutions wishing to highlight a particular character.

I named this project "Pupilo Books" because it began with my daughter’s toy, Pupilo. Currently, I’m creating these books for family and friends, but I’ve noticed genuine excitement from others who have expressed interest in having one. To make this accessible, I’ve set up an email where you can reach out to bring your vision to life: pupilobooks@gmail.com.

Need help with automation, AI, or data science? Let’s talk.

Book a 20-min consultation

Conclusion

Currently, the internet is populated with AI-generated books, but many of the examples I’ve come across lack a personal, humorous touch. There’s no one better than you to create an emotional or funny story about your mother, it’s not something a company can replicate. You need to provide the inputs, and that’s the concept behind Pupilo Books.

That said, there are still limitations with current APIs. For example, generating images of public figures is challenging, and characters’ appearances, such as clothing, hairstyles, or body types, often change inconsistently throughout the story. This makes it difficult to maintain coherence or introduce a second character with proper context.

For now, Pupilo Books can produce funny and emotional stories, but they lack precision, so the inputs must follow specific constraints, and users should understand the limitations.

Even with these challenges, experimenting with AI is incredibly fun and a great creative exercise. It encourages you to explore state-of-the-art models and think outside the box. Given the fierce competition among AI companies, simply exploring and experimenting in your free time can be already very rewarding and prepare you for a future where AI is very much like oxygen.