Building a web agent that can directly control a personal computer device to automate tasks is significantly more challenging than building a traditional function-calling agent. Here's why:

• Complexity of Tasks: Even a simple task like sending an email requires multiple steps for a web agent - opening the Gmail website, clicking the search bar, typing the email, clicking the reply button, and finally clicking send. Each of these steps has the potential for errors, requiring stronger memory and reasoning abilities from the agent.

• Interface Understanding: The agent needs to accurately understand the user interface, whether by parsing the HTML/XML structure or analyzing screenshots using computer vision techniques. Extracting the relevant information and deciding on the next action to take is a complex challenge.

• Positioning Accuracy: Precisely locating the correct UI elements to interact with, such as buttons or input fields, is crucial for the agent's success. Techniques like using OCR and combining multiple models have shown promise, but this remains a significant hurdle.

• Speed and Efficiency: The nature of this type of agent, going through multiple steps for even simple tasks, inherently makes it less efficient than traditional agents. Improving the speed and overall task completion rate is an important goal.

• Accuracy and Reliability: Ensuring the agent can accurately perform tasks without getting stuck in infinite loops or making mistakes is critical for real-world applications. Addressing these accuracy and reliability challenges is a key focus area.

Despite these challenges, the potential benefits of a web agent that can handle a wide range of personal and work-related tasks are significant. Ongoing research and development in areas like computer vision, language models, and task planning are helping to advance the state of the art in this field.

There are three main approaches that web agents use to understand and interact with user interfaces:

1. HTML/XML-based Approach:

   • The agent extracts the HTML or XML structure of the website and uses this information to understand the layout and interactive elements.
   • The agent can then use this knowledge to locate and interact with specific UI elements, such as input fields, buttons, and links.
   • This approach is relatively mature, but it has limitations in handling complex or poorly structured websites.

2. Vision-based Approach:

   • The agent uses computer vision models to analyze screenshots or images of the user interface.
   • This allows the agent to identify and locate UI elements, even in the absence of clean HTML/XML data.
   • Techniques like Saliency Mapping and Optical Character Recognition (OCR) are used to pinpoint the exact coordinates of interactive elements.
   • Combining vision models with language models (e.g., GPT-4) can improve the accuracy of this approach.

3. Hybrid Approach:

   • This combines the strengths of the HTML/XML-based and vision-based approaches.
   • The agent uses both the structured data from the website and the visual information from screenshots to understand the interface.
   • This approach can handle a wider range of website structures and provide more accurate interaction with UI elements.

The key challenges in building effective web agents include:

1. Speed: The multi-step nature of web interactions can make web agents slower than traditional function-calling agents.
2. Accuracy: Precisely locating and interacting with UI elements is a complex task that requires advanced computer vision and language understanding capabilities.
3. Task Completion: Maintaining context and avoiding infinite loops are important for ensuring web agents can successfully complete complex tasks.

Despite these challenges, web agents have the potential to unlock a wide range of use cases, particularly in areas like web scraping, where their ability to interact with any website can be highly valuable. Projects like WebQL are making it easier to build these types of universal web agents.

Firstly, one thing I quickly realized is that this type of web, mobile, or desktop agent that can directly control the personal computer device is multiple magnitudes harder than normal function-calling agents we're building. Let's say we're building a simple inbox manager agent who can perform actions like sending an email. With a normal function-calling agent, all you need to do is call a predefined function called "send email" and pass on the email content, and the task is done. There's not much room for error.

However, if we try to get a web agent to complete the simple task of sending an email, it will need to go through at least four different steps. It will firstly need to open Gmail.com in the web browser, click the search bar, search for the specific email to reply to, click on the right email from the search result, click on the reply button, type the response, and click Send. So it takes a lot more steps to complete even a basic task. There's a lot more room for error in this process, as any of those steps the agent can get wrong. The agent also needs stronger memory and reasoning ability to remember what it has done before to avoid repeating the same errors.

So, in short, it is a lot more challenging to build, but if this ability is achieved, it is super exciting and opens up opportunities for multiple huge markets.

How does this system actually work? There are three main common ways we can approach this:

1. HTML or XML-based approach: We'll try to extract the HTML file of each website and give those HTML DOM elements to the agent as context, so the agent will be able to understand the structure of the website and then decide what to do next. This is the most mature method, but it has limitations, such as not being able to handle tasks involving images.

2. Vision-based approach: Instead of feeding the agent the original HTML code, we can take a screenshot and send it to a multimodal model, where it can understand, reason, and plan the next step. The hardest part of this approach is accurately locating the exact UI element to interact with.

3. Hybrid approach: Some teams have combined the strengths of both HTML/XML-based and vision-based approaches, using a combination of language models and optical character recognition (OCR) to improve accuracy.

Overall, there are three main challenges with these web and mobile desktop agents: speed, accuracy, and task completion. However, despite these limitations, we can still build useful tools with this web agent approach, particularly in the area of web scraping, where a universal API to access any website content can be extremely valuable.

One project that has shown promise in this area is WebQL, which is designed specifically to solve the problem of finding and locating UI elements for agents to interact with. By using WebQL, we can create a universal e-commerce product information scraper that can work across different websites, simply by changing the URL and a few variables.

The possibilities with these multimodal approaches are exciting, and I'm looking forward to seeing what kind of interesting web or mobile agents the community starts building. If this topic interests you, please let me know, and I'll be happy to create a more in-depth video on it.

The development of web agents that can directly control personal computer devices to automate tasks is a complex challenge, with several key problems that need to be overcome:

1. Speed: The nature of this type of agent requires going through multiple steps to complete even simple tasks, making them inherently less efficient compared to traditional function-calling agents.

2. Accuracy: Accurately locating and interacting with specific UI elements on websites and applications is a significant challenge. Approaches like using HTML/XML structure, multimodal models, and combinations of techniques like OCR and CLIP have shown progress, but there is still room for improvement.

3. Task Completion: Web agents can often get stuck in infinite loops, forgetting the steps they've taken before and repeatedly encountering the same problems. Resolving this issue of maintaining context and task completion is crucial for increasing the adoption of these agents.

To address these problems, several techniques and tools have been explored:

• HTML/XML-based Approach: Extracting and cleaning up the HTML structure to provide the agent with a more manageable context has shown promise, but is limited in handling tasks involving images and poorly designed websites.

• Multimodal Approach: Using computer vision techniques like screenshot analysis, OCR, and CLIP to understand the UI and locate interactive elements has improved accuracy, but still faces challenges with complex or condensed interfaces.

• Specialized Models: Projects like Cook Agent, a visual language model designed specifically for understanding and interacting with GUI screenshots, have demonstrated better performance in web and mobile task completion.

• Web QL: This open-source library provides a way to easily define queries to locate and interact with UI elements, simplifying the process of building accurate web agents.

By leveraging these techniques and tools, developers can start building powerful web agents that can handle a wide range of web-based tasks, from web scraping to complex workflows involving multiple websites and applications. The key is to find the right balance of approaches to address the speed, accuracy, and task completion challenges.

One of the key challenges in web scraping has been the need to maintain custom scrapers for each website, as their structure and layout often change over time. However, the emergence of web agents that can directly control the user interface of a web browser opens up new possibilities for building more universal and robust web scrapers.

These web agents leverage advanced AI models, such as large language models and computer vision techniques, to understand and interact with web interfaces in a more human-like manner. By simulating real user interactions like clicking, scrolling, and typing, these agents can navigate and extract data from a wide range of websites without the need for custom code.

One such open-source project, called WebQL, provides a powerful tool for building these web agents. WebQL allows you to define queries that specify the UI elements you want to interact with, such as input fields, buttons, and product listings. The library then uses computer vision and other techniques to accurately locate and interact with these elements, even on complex and dynamic websites.

Using WebQL, you can quickly build universal web scrapers that can be applied to different e-commerce sites, for example, to extract product information like name, reviews, price, and shipping details. The same script can be reused across multiple sites, significantly reducing the maintenance overhead compared to traditional web scraping approaches.

Beyond web scraping, these web agents can also be used to automate a wide range of web-based tasks, such as booking flights, managing email, and interacting with productivity tools. By combining the flexibility of these agents with the accuracy and reliability of libraries like Playwright, developers can create powerful personal assistants that can handle complex, multi-step workflows on the web.

While there are still some challenges to overcome, such as ensuring reliable task completion and improving the speed and accuracy of the agents, the potential of this technology is clear. As the underlying AI models and integration techniques continue to evolve, we can expect to see more and more innovative applications of web agents in the near future.

To build a universal e-commerce scraper using WebQL, we'll follow these steps:

1. Install the required libraries:

   • pip install webql
   • pip install playwright

2. Set up the WebQL API key in an .env file.

3. Create a Python script called ecommerce_scraper.py with the following code:

import os
from dotenv import load_dotenv
from webql import WebQL
from playwright.sync_api import sync_playwright
import csv
import json

load_dotenv()

def save_json_to_csv(data, filename):
    columns = ['product_name', 'num_reviews', 'price', 'rating']
    with open(filename, 'w', newline='') as csvfile:
        writer = csv.DictWriter(csvfile, fieldnames=columns)
        writer.writeheader()
        for product in data['results']['products']:
            writer.writerow({
                'product_name': product['name'],
                'num_reviews': product['num_reviews'],
                'price': product['price'],
                'rating': product['rating']
            })

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False)
    page = browser.new_page()

    webql = WebQL(os.getenv('WEBQL_API_KEY'))
    session = webql.create_session(page)

    home_query = {
        'search_box': 'input[name="q"]',
        'search_button': 'button[type="submit"]'
    }
    home_elements = session.query(home_query)
    home_elements['search_box'].fill('coffee machine')
    home_elements['search_button'].click()

    search_query = {
        'products': ['name', 'num_reviews', 'price', 'rating']
    }
    search_results = session.query(search_query)

    save_json_to_csv(search_results, 'ecommerce_products.csv')

    browser.close()

4. Run the script:

python ecommerce_scraper.py

This script will:

1. Open a browser session using Playwright.
2. Use WebQL to locate the search box and search button on the homepage, fill in "coffee machine", and click the search button.
3. Use WebQL to extract the product information (name, number of reviews, price, and rating) from the search results page.
4. Save the extracted data to a CSV file named ecommerce_products.csv.

The key aspects of this implementation are:

• Using WebQL to define the UI elements to interact with on the website.
• Leveraging WebQL's ability to extract structured data from the website.
• Saving the extracted data to a CSV file for further processing or analysis.

This approach allows you to create a universal e-commerce scraper that can work across different e-commerce websites, as long as they have a similar structure. By modifying the WebQL queries, you can adapt the scraper to extract different types of data from various e-commerce platforms.

Firstly, one thing I quickly realized is that this type of web, mobile, or desktop agent that can directly control the personal computer device is multiple magnitudes harder than normal function-calling agents we're building. The reason is that even for a simple task like sending an email, the web agent needs to go through multiple steps - opening Gmail.com, clicking the search bar, searching for the email, clicking the reply button, typing the response, and clicking send. This process has a lot more room for error, and the agent needs stronger memory and reasoning abilities to remember and not repeat the same mistakes.

To make this system work, there are a few common approaches. The first is the HTML or XML-based approach, where the agent is given the structure of the website's HTML DOM elements to understand the interface and decide on the next actions. This method is the most mature, but it has limitations, especially for tasks involving images or poorly-designed websites.

The second approach is the vision-based method, where the agent uses a multimodal model to understand a screenshot of the interface and provide instructions on where to interact. This method has seen significant progress, with techniques like Saliency Maps and the combination of GPT-4 and OCR models improving accuracy. The open-source model called Cook Agent, designed specifically for understanding and interacting with GUI screenshots, is a promising example of this approach.

Despite the progress, there are still three main challenges with these web agents: speed, accuracy, and task completion. The nature of these agents makes them inherently less efficient than traditional function-calling agents, and accuracy is still an issue, especially for complex interfaces. Task completion is also a problem, as the agents can sometimes get stuck in infinite loops.

However, there are still many useful applications for these web agents, particularly in the area of web scraping. Traditional web scraping has been a manual and error-prone process, as each website has a different structure. With a universal web agent that can learn to interact with any website, companies can build more robust and scalable web scrapers.

One tool that can help with this is WebQL, an open-source project designed to solve the problem of accurately locating UI elements for the agent to interact with. WebQL provides a simple API that allows you to define the elements you want to interact with, and it returns the exact DOM elements, making it much easier to build accurate web agents.

To demonstrate this, I've included an example of how you can use WebQL to build a universal e-commerce product scraper. The script can navigate to any e-commerce website, search for a product, and extract the relevant information (product name, reviews, price, rating, and shipping fee) into a CSV file. This same script can be used for different e-commerce sites by simply changing the URL and file name.

Additionally, I've included a more complex example where the agent can extract Tesla pricing and delivery information from multiple countries, format the data in a Google Sheet, and automatically share the results in Slack. This showcases the power of these web agents and the potential for automating complex workflows.

In conclusion, while building these web agents is a significant challenge, the potential benefits are substantial. By overcoming the limitations of speed, accuracy, and task completion, these agents can unlock a wide range of use cases, from web scraping to automating complex personal and business tasks. With the help of tools like WebQL, the development of these agents is becoming more accessible, and I'm excited to see what developers will create in this space.

This new type of agent that can directly control personal computer devices to automate tasks is a fundamentally different approach compared to traditional function-calling agents. Instead of pre-defining specific functions for each website or application, this agent aims to learn fundamental skills like mouse clicks, scrolling, and typing, allowing it to interact with any new website or application.

The key challenges in building such an agent include ensuring speed, accuracy, and task completion. While the initial approaches using HTML/XML parsing and multi-modal models have had some success, the emergence of specialized models like CookAgent shows promising progress in addressing these challenges.

One area where this technology can be particularly useful is web scraping. By creating a "universal API" to access any website, these agents can significantly reduce the maintenance overhead for companies that need to regularly scrape competitor websites or aggregate data from various sources.

The example provided demonstrates how to build a versatile e-commerce product scraper using the WebQL library, which simplifies the process of locating and interacting with UI elements. This approach can be extended to more complex workflows, such as extracting Tesla pricing and delivery information across different countries, formatting the data in a Google Sheet, and automatically sharing the results in Slack.

Overall, this new type of agent represents an exciting development in the field of AI, with the potential to unlock a wide range of practical applications by automating various personal and work-related tasks.