Imagine owning an e-commerce store with countless tags and metrics, endless work, and frequent new skills needed to maintain success.
What if there were an AI solution to automate repetitive tasks, assist with problem-solving, and help run the business?
Merchants often wasted time searching through scattered help documents just to find simple answers to learn basic skills.
They also had to take action in the store to make the desired changes. There are wait times for Shopify’s human support agents who have limited permissions within a store.
That’s why the most well-known e-commerce platform, Shopify, has introduced Sidekick, an “AI-enabled commerce assistant.”
This agent can use info from relevant Shopify help documents to help people solve problems. The key differentiator of this AI from a typical LLM like ChatGPT is that it can complete nearly any action in your store for you and is built around e-commerce.
Before we explore how it helps businesses, let’s learn how it works. To do that, we need to go back to the launch of Shopify Magic.
Shopify Magic Enhances Basic Tasks
Debuting as a suite of AI-powered productivity features, Shopify Magic shocked the e-commerce world.
Magic was originally a fixed set of features integrated into a limited set of specific areas of a Shopify store. A key example is writing product descriptions. The merchant explains key product details, selects the preferred tone, and provides special instructions as needed.
Afterwards, once the approved description is imported, there is a continued editing with Magic button to rephrase, extend, or simplify. This feature also incorporated a “try again” button to regenerate the changes.
This e-commerce wizard is made possible by combining Shopify’s help documents with publicly available LLMs like ChatGPT to get the best reasoning and generation alongside the most accurate Shopify information.
Since LLMs may hallucinate or produce biased content, merchants always approve changes before allowing the agent to take action.
Magic was the precursor and testing grounds for Shopify’s agent, Sidekick. It allowed them to experiment with the practicality of introducing LLMs to the platform and see how users would react.
Sidekick Can Do Almost Anything
Sidekick can help with almost any topic covered in Shopify’s help docs.
For example, if you had a question about the specific details of Shopify’s discounted shipping rates through its label purchasing portal, Sidekick could walk you through what the discount’s terms are and how to use it.
It can also see where the merchant is in the Shopify store backend, so contextually, based on where you are, it can help you navigate to the current pages based on what you want.
The above shipping question is answered based on Shopify help docs, with the LLM generating the text. However, when the agent analyzes data, it also uses the LLM to reason to explain how the business can problem-solve and the documents to provide definitions and context for each metric and issue.
Sidekick can also take action on its own, with human approval and in accordance with its guardrails. These include moving money between accounts and checking balances, generating content, images, and text, and segmenting customers. It can also generate analytics reports from a text prompt, create email campaigns, and toggle settings.
It can do anything, right? Well, it has a few key limitations:
- Sidekick can’t talk to your customers or handle support conversations on your behalf.
- Sidekick can’t share private information or data from other Shopify merchants with you.
- Sidekick is never allowed to make changes to your shop without your approval, and will only present options for you to review and approve.
This agent is unique because it can help with nearly anything in a Shopify store. A key factor in making this possible is the agentic loop.
The Agentic Loop Makes Sidekick So Impressive
According to Shopify, it is built on what Anthropic calls the “agentic loop”. Let’s break that down.
First, a human gives a prompt to an LLM, which processes it and decides which actions to take.
Then the agent takes action in its environment, in this case, the Shopify ecosystem.
Feedback is gathered from the human; if positive, it completes the cycle, stopping it; if constructive and requesting further changes, the cycle repeats, building on the original prompt.
The reason it is an “agentic loop” is that the cycle continues until the user approves the actions; otherwise, it continues looping.
In practice, this agentic loop setup allows Sidekick to handle such a variety of requests. Below are some screenshots from an example conversation with the agent in a Shopify store.
As seen in the screenshots, it processes the request with the LLM first, as indicated by sentences starting with “I’ll help” or “I’ll get”.
The agent then takes action, segmenting customers from the store data to give me the answer, presenting it in text and a clickable box that opens the analytics menu for further data.
Lastly, the agent seeks feedback in two ways: the thumbs-up/down buttons report data to Shopify, and keeping the chat open without changing the user’s page.
These functions make it easy to send follow-up instructions to continue the cycle if needed.
The agent can also compare and analyze reasons for metric changes:
This time, I continued the loop and asked a follow-up question.
The agent provided a lot of relevant data and a clear key point to improve at the end of the response. It created an analytics report to gather data on “Marketing Campaign Effectiveness” with sessions, conversion rates, traffic sources, and cart/checkout metrics.
The agent also explored marketing data, such as the number of emails sent and the content in those emails.
Most notably, the recommendations have four key pillars of methods to increase traffic, which is the overall fix to the sales drop, further emphasized in the final words of the response, “you just need more visitors”.
In practice, the agentic loop is very effective in problem-solving, making it powerful. Using a human prompt, LLM processing, agentic action, and allowing follow-up feedback, this setup is capable.
But what happens if the agent’s tools start interfering with one another?
Scaling AI Requires Just-in-Time Instructions
As agents are built, a common problem is scaling a few well-defined tools to a massive amount of specialized capabilities. Needless to say, Shopify ran into this issue as well. A single agent like Sidekick can’t actually do it all.
Within the agent is a variety of other identities, essentially specialty tools. The issues occur when the agent combines tools, giving unexpected outcomes, and when there are multiple ways to accomplish the same task. Each tool is an identity, so as one grows, so do the others.
JIT Instructions assign identities before the agent executes tasks. Using identity orchestration software gives the agent credentials, a digital permission to do a specific thing. It’s like giving an employee a temporary ID badge for one shift. The credentials expire, and the identity is retired from the agent after the task is completed.
This eliminates tool combination and selection issues because the agent has access only to the permissions needed for the task at hand.
As Shopify says, “not a token less, not a token more,” because the identity gives the LLM the perfect context to pick the right tools. This allows Shopify to get faster, more consistent results, cut costs, and increase security.
Assigning identities to tools prevents the model from accidentally taking a completely wrong action or mixing up tools.
Human Judge-Based Datasets Are Powerful
Shopify’s developers focused on building their own Ground Truth Sets rather than using existing “golden” datasets. This had a few steps:
First, Shopify experts would label and categorize merchants’ conversations with their human support agents.
Then they would view the question and evaluate the appropriate responses using Cohen’s Kappa, Kendall’s Tau, and Pearson’s Correlation to measure inter-annotator agreement.
Finally, they would use the human agreement levels with the LLM’s response in a practice conversion as the theoretical maximum “correctness” the LLM could achieve. Correctness involves selecting the right tools to complete the task, which the human support agent in the past would do themselves.
Let’s briefly explore Cohen’s Kappa, Kendall Tau, and Pearson’s Correlation. Each of the three methods measures agreement between defined entities based on the type of data received.
The data is classified as nominal, ordinal, or metric.
Nominal data has no natural ranking, and all the pieces can be obviously separated. E.g. blonde hair vs brown hair vs black hair.
Ordinal data has the same characteristics as nominal data in that features can be distinguished, but they also have a natural order/ranking; for example, grades. An A is better than a B, but the difference between the two isn’t necessarily always the same or known.
Metric data has measurable differences between the values of each item. E.g. Height is increasing by a certain amount, and weight is also increasing by a certain amount:
Height: 150, 160, 170 (differences of 10).
It could also look like:
Height: 151, 165, 173
The second example above has multiple differences.
Metric data can be further divided into ratio and interval scales. On an interval scale, only how much more or less one thing is from another can be said; however, on the ratio scale, a percentage can be calculated of the difference between two things.
Each of these allows one to quantify the agreement between people, some more precise than others. Using all three principles enables the AI to be trained on a variety of real agreement data and calculate accurate correctness expectations for the LLM.
Cohen’s Kappa measures how often raters agree while adjusting for chance-based agreement.
Kendall’s Tau measures how similar the rankings of things are.
Pearson’s Correlation is a linear representation of the strength and direction of the differences between data points.
Above is an example of a Pearson’s correlation graph; it shows a strong correlation as all the points are close to each other.
Shopify can measure the agreement of the human judge’s thoughts on the support agent’s responses relative to something else.
With a graph like this, by setting the x-axis to be a number 1–10 representing the strength with which a human judge agrees with the support agent giving a correct response, the higher the number, the more they agree.
In the prior example, the y-axis could be the number of Shopify help documents analyzed before answering. What the y-axis represents depends on what the Shopify developers want to measure; however, it is common that one axis will measure agreement.
Shopify may use a correlation to measure human agreement levels relative to something else; in our example, it is compared to help document reading.
This data is crucial for understanding how correct the LLM can be. Our example would allow Shopify developers to develop a range of how many help documents the AI agent should read.
Too few documents would make the answer insufficiently detailed, and too many may confuse the agent.
Measuring agreement levels allows Shopify to build the agent to be as correct as possible by helping it craft optimal dataset usage.
LLM Judges and Simulations Effectively Train Models
Shopify built an AI-powered merchant simulator that captures the goals of real merchant support conversations, replays them into multiple tools, checks which one provides the best response, and trains the AI to use that tool to achieve that specific goal.
While developing the agent, Shopify built a specialized LLM that analyzes generated text and creates feedback to train it continuously.
As Sidekick improves and training continues, the goal is for the agent to generate responses that the human and the LLM would both give similar critiques on. Shopify’s goal was to be able to swap the provided feedback, and when it is given, the agent would have no idea who wrote it.
Both tools, the simulation and the LLM judge, are used beneath the surface while Sidekick generates a response. It checks which tool to use and then evaluates the response.
If the response is poor, it will regenerate it, possibly with a different tool if the LLM assigned the wrong identity (remember JIT instructions).
LLM judges help Shopify ensure the agent is making a good decision.
Retraining Can Stop Reward Hacking Code Breaks
Sidekick was built on using LLM judges as its reward signals, and syntax and schema validation.
Syntax errors occur when code is not properly written; invalid code cannot run, which causes the model to break.
Schema issues occur when the expected output doesn’t align with the input requirements. This can also break the code if the output depends on requirements it doesn’t have, as the code won’t align.
For example, if requested to respond with a name, age, and email, but the code prior only defines the name and age, there will be a validation error as the scripts defining the response mention the email field, but it is never established prior.
These can also cause errors in the user’s anticipated response. If the code doesn’t break from the setup, the user may receive unexpected results.
- Opt-out Hacking: Instead of attempting difficult tasks, the model would explain why it couldn’t help
- Tag Hacking: Using customer tags as a catch-all instead of proper field mappings
- Schema Violations: Hallucinating IDs
The model had to be retrained in specific areas where it originally struggled to fix these issues. These initially caused new errors, but over time, the syntax accuracy rose to 99% after this process.
Retraining in this area also required improving the LLM judges to more consistently recognize these failures, preventing the error from reaching the merchant. This led to a 9% increase in agreement with human evaluation of the response.
Fixing code breaks is essential in agent development for a reliable model.
AI in E-commerce Fuels Creativity
The development of agents like Sidekick will not make human skills invaluable, replace jobs in e-commerce businesses, or make billionaires overnight.
It will increase efficiency and free up merchants’ time as new ideas can be executed more easily.
In my opinion, the best use case is developing completely new, one-of-one sections in your theme. Shopify themes have limitations in what can be created without coding skills. The agent eliminates this barrier, allowing people to build more unique stores and turn their creative ideas into reality by coding sections for the merchant.
These sections contain built-in no-code customizations allowing users to edit the padding, content colors, etc.
In the future, this means easier development of extremely creative Shopify stores and will reduce merchant frustration with completing tedious actions, especially in settings and analytics.
When these kinds of agents are common across all major e-commerce platforms, the user experience and simplicity will be the key factors in what merchants prefer.
Thank you for reading, and please connect to my LinkedIn if you would like to chat about AI, e-commerce, or something else cool!
Sources
- Building Production-Ready Agentic Systems — Shopify Engineering
- Shopify Sidekick Is Like ChatGPT but for E-Commerce Merchants — TechCrunch
- AI Product Descriptions — Shopify Blog
- Shopify Magic — Shopify Help Center
- Shopify Sidekick — Shopify Help Center
- Just-in-Time Provisioning Creates Artificial Agent Identities on Demand — Strata AI
- Cohen’s Kappa — Numiqo
- Level of Measurement — Numiqo
- LLM as a Judge — Evidently AI