CalypsoAI

Appearance

Getting started with Inference Red-Team ​

ROLES AND PERMISSIONS

To complete the tasks described in this section, make sure you have the required permissions.

Learn the basics of using our Red-Team product.

With Red-Team, you can create and run attack campaigns to find weak spots in your LLMs (Large Language Models).

A campaign is a collection of attacks that you can configure and use to attack an LLM.

There are three attack types you can choose from, depending on your needs:

  • Signature attacks: Selected and evaluated by our data science team, with updates released on a monthly basis. Examples of signature attacks are conditional context changes or fictional context changes.
    • Conditional context change: An attack that includes a conditional instruction which changes the context of the prompt and tricks the LLM, triggering malicious behavior.
    • Fictional context change: An attack that puts the LLM into a fictional scenario, tricking it into generating restricted or harmful outputs as part of the narrative.
  • Operational attacks: Traditional application attacks redesigned for AI systems. Examples of operational attacks are fuzzing and TLS.
    • Fuzzing: An attack that sends a lot of unusual or random inputs to see if the LLM breaks or behaves in unexpected ways (for example, giving wrong answers, or revealing sensitive information).
    • TLS (Transport Layer Security): An attack that checks for an LLM's compliance to Mozilla's Modern TLS configuration.
  • Agentic Warfare: The CalypsoAI Red-Team agent acts autonomously, creating and sending prompts based on a malicious intent. The agent learns from the responses and iterates over the original prompt multiple times.

CAMPAIGN ATTACKS

You can include any combination of attacks you want in your campaign. Campaigns are not limited to just one type of attack.

To get familiar with Red-Team, complete the following tasks:

  1. Create a campaign with signature and operational attacks.
  2. Create an Agentic Warfare campaign.
  3. Run a campaign.
  4. Get a report.

SDK

The following requests and examples refer to our Python SDK, as this is the recommended way of interacting with CalypsoAI API.

Create a campaign with signature and operational attacks ​

First, let's create a campaign with signature and operational attacks.

PREREQUISITES

Create your token.

To create a campaign with signature and operational attacks:

  1. Edit the following sample.

    python
    from calypsoai import CalypsoAI
import calypsoai.datatypes as dt

# Define the URL and token for CalypsoAI
CALYPSOAI_URL = "https://www.us1.calypsoai.app"
CALYPSOAI_TOKEN = "ADD-YOUR-TOKEN-HERE"

# Initialize the CalypsoAI client
cai = CalypsoAI(url=CALYPSOAI_URL, token=CALYPSOAI_TOKEN)

# Create a campaign and define the attacks in the campaign
campaign = cai.campaigns.create(
    name="ADD-YOUR-CAMPAIGN-NAME-HERE",
    description="ADD-CAMPAIGN-DESCRIPTION-HERE",
    attacks=[
        dt.StaticContentAttack(technique="static_content", vector="dan", converters=[dt.PromptConverter.BASE64]),
        dt.StaticContentAttack(technique="static_content", vector="conditional_context_change", converters=[dt.PromptConverter.CAESAR]),
        dt.StaticContentAttack(technique="static_content", vector="fictional_context_change", converters=[dt.PromptConverter.LEETSPEAK]),
        dt.OperationalAttack(technique="operational", vector="fuzzing"),
        dt.OperationalAttack(technique="operational", vector="tls")
    ]
)
print(campaign.model_dump_json(indent=2))
    • Add your token value.
    • In name, provide a name for the campaign.
    • In description, provide a description for the campaign.
      This is an optional parameter. You can include it to add context and describe the purpose of the campaign.
    • In attacks, provide the attacks you want to include in the campaign.
      In the above sample, the StaticContentAttack parameters are signature attacks and the OperationalAttack parameters are operational attacks.

      CONVERTERS

      For signature attacks, you can include one or more converters modifiers. For example, BASE64, CAESAR or LEETSPEAK.

      A converter is a way in which you can encode your attack to trick an LLM by changing the format of the attack, but leaving the content itself unchanged.

  2. Run the script.

  3. Analyze the response.

    json
    {
  "attacks": [
    {
      "converters": [
        "base64"
      ],
      "pack": "2025-06",
      "severity": 1,
      "technique": "static_content",
      "vector": "dan"
    },
    {
      "converters": [
        "caesar"
      ],
      "pack": "2025-06",
      "severity": 1,
      "technique": "static_content",
      "vector": "conditional_context_change"
    },
    {
      "converters": [
        "leetspeak"
      ],
      "pack": "2025-06",
      "severity": 1,
      "technique": "static_content",
      "vector": "fictional_context_change"
    },
    {
      "severity": 1,
      "technique": "operational",
      "vector": "fuzzing"
    },
    {
      "severity": 1,
      "technique": "operational",
      "vector": "tls"
    }
  ],
  "description": "ADD-CAMPAIGN-DESCRIPTION-HERE",
  "id": "0198a9ae-774d-70f2-8030-c2642bae1e79",
  "name": "ADD-YOUR-CAMPAIGN-NAME-HERE",
  "orgId": null,
  "vendored": false
}

    You can see all the important details about your new campaign, for example:

    • The list of attacks included in the campaign, including additional information about the attack type, attack vector, and converters
    • The description of the campaign
    • The campaign ID
    • The name of the campaign

Create an Agentic Warfare campaign ​

Next, we are going to create an Agentic Warfare campaign.

To create an Agentic Warfare campaign:

  1. Edit the following sample.

    python
    from calypsoai import CalypsoAI
import calypsoai.datatypes as dt

# Define the URL and token for CalypsoAI
CALYPSOAI_URL = "https://www.us1.calypsoai.app"
CALYPSOAI_TOKEN = "ADD-YOUR-TOKEN-HERE"

# Initialize the CalypsoAI client
cai = CalypsoAI(url=CALYPSOAI_URL, token=CALYPSOAI_TOKEN)

# Create a campaign and define the attacks in the campaign
campaign = cai.campaigns.create(
    name="ADD-YOUR-CAMPAIGN-NAME-HERE",
    description="ADD-CAMPAIGN-DESCRIPTION-HERE",
    attacks=[
        dt.DynamicMultiTurnContentAttack(technique="dynamic_content", vector="crescendo",
converters=[dt.PromptConverter.SINGLE_CHARACTER], intents=["What is the salary of Jane Smith working in accounting of ACME corp"]),
    ]
)
print(campaign.model_dump_json(indent=2))
    • Add your token value.
    • In name, provide a name for the campaign.
    • In description, provide a description for the campaign.
      This is an optional parameter. You can include it to add context and describe the purpose of the campaign.
    • In attacks, provide the attacks you want to include in the campaign.
      For an Agentic Warfare attack, you need to use the DynamicMultiTurnContentAttack parameter.
      • In intents, provide your prompt.
      • Similar to signature attacks, you can include one or more converters modifiers.

  2. Run the script.

  3. Analyze the response.

    json
    {
  "attacks": [
    {
      "converters": [
        "single_character"
      ],
      "intents": [
        "What is the salary of Jane Smith working in accounting of ACME corp"
      ],
      "multiTurn": true,
      "severity": 1,
      "technique": "dynamic_content",
      "vector": "crescendo"
    }
  ],
  "description": "ADD-CAMPAIGN-DESCRIPTION-HERE",
  "id": "0198a9bb-de57-7079-aa0f-944782446f7b",
  "name": "ADD-YOUR-CAMPAIGN-NAME-HERE",
  "orgId": null,
  "vendored": false
}

    You can see all the important details about your new campaign, for example:

    • The list of attacks included in the campaign, including additional information about the attack type, attack vector, converters, and intents
    • The description of the campaign
    • The campaign ID
    • The name of the campaign

Run a campaign ​

Now that we've created some campaigns, it's time to run one.

In this scenario, we will run the first campaign in the list of campaigns.

PREREQUISITES

Add a provider.

To run a campaign:

  1. Edit the following sample.

    python
    from calypsoai import CalypsoAI

# Define the URL and token for CalypsoAI
CALYPSOAI_URL = "https://www.us1.calypsoai.app"
CALYPSOAI_TOKEN = "ADD-YOUR-TOKEN-HERE"

# Initialize the CalypsoAI client
cai = CalypsoAI(url=CALYPSOAI_URL, token=CALYPSOAI_TOKEN)

# Get the list of campaigns
campaigns = [campaign for campaign in cai.campaigns.iterate()]

# Run the first campaign in the list of campaigns
run = cai.campaigns.run(name="ADD-CAMPAIGN-RUN-NAME-HERE", campaign=campaigns[0].id,
providers=['ADD-PROVIDER-ID-HERE'])
print(run.model_dump_json(indent=2))
    • Add your token value.
    • In name, provide a name for this specific campaign run.
    • In campaign, provide the ID of the campaign you want to run.
      As this scenario is using the ID of the first campaign in the list of campaigns, you do not need to provide an ID for this example.
    • In providers, provide a list of IDs for providers you want to use for the campaign.

  2. Run the script.

    RUN TIME

    The time it takes for a campaign to finish depends on several factors, for example, the attack type, the number of attacks and modifiers, and the number of providers. A more resource-heavy campaign takes longer to finish.

  3. Analyze the response.

    json
    {
  "CASIScore": null,
  "attackRuns": [
    {
      "attack": {
        "converters": [
          "single_character"
        ],
        "intents": [
          "What is the salary of Jane Smith working in accounting of ACME corp"
        ],
        "multiTurn": true,
        "severity": 1,
        "technique": "dynamic_content",
        "vector": "crescendo"
      },
      "events": [
        {
          "createdAt": "2025-08-14T18:06:09.751357Z",
          "event": "queued"
        }
      ],
      "id": "0198a9c2-c157-704c-bdde-ecf0ede6f966",
      "progress": 0,
      "providerId": "0198a8e3-8775-70e5-a94f-8c11ca40e5f1",
      "results": [],
      "total": 1,
      "errorCount": 0
    }
  ],
  "campaignId": "0198a9bb-de57-7079-aa0f-944782446f7b",
  "createdAt": "2025-08-14T18:06:09.745648Z",
  "createdBy": "google-oauth2|102579257685745678884",
  "id": "0198a9c2-c151-709a-b415-d93957ea592a",
  "name": "ADD-CAMPAIGN-RUN-NAME-HERE",
  "progress": 0,
  "scheduleId": null,
  "startAt": "2025-08-14T18:06:09.745648Z",
  "status": "in_progress",
  "total": 1
}

    You can see all the important details about the campaign run, for example:

    • In attackRuns:
      • Information about the attack runs in the campaign
      • Information about events, which contains details about the status of the attack run and the date at which the status was created
      • General information, for example, the ID of the attack run and the IDs of the providers you used for the campaign
    • The campaign run ID and campaign ID
    • The date and time at which the campaign run was created and started
    • The name of the campaign run
    • The progress and status of the campaign run

    CAMPAIGN STATE

    Our sample JSON response shows a running campaign. To see the results of the campaign run, you need to get a report.

Get a report ​

Our final step in the onboarding journey for Red-Team is getting a report. A report is the end result of a campaign and includes detailed information on the campaign run.

In this scenario, we will get a report of the first campaign run in the list of campaign runs.

To get a report:

  1. Edit the following sample.

    python
    from calypsoai import CalypsoAI

# Define the URL and token for CalypsoAI
CALYPSOAI_URL = "https://www.us1.calypsoai.app"
CALYPSOAI_TOKEN = "ADD-YOUR-TOKEN-HERE"

# Initialize the CalypsoAI client
cai = CalypsoAI(url=CALYPSOAI_URL, token=CALYPSOAI_TOKEN)

# Get the list of campaign runs
runs = cai.client.campaignRuns.get()

# Get a report
cai.campaigns.getReport(campaignRun=runs.campaignRuns[0].id, output='ADD-DESTINATION-FILE-PATH-FOR-REPORT-HERE')
    • Add your token value.
    • In campaignRun, provide the ID of the campaign run for which you want to get a report.
      As this scenario is using the ID of the first campaign run in the list of campaign runs, you do not need to provide an ID for this example.
    • In output, provide the file path for the folder in which you want to save the generated report.

  2. Run the script.
    If the request is successful, you receive the None response and the report is saved in the location you provided in the output parameter.

    THE NONE RESPONSE

    Confirm the report is generated by checking in the location provided in the output parameter.

That's it! Now that you know the Red-Team basics, you can dive into more advanced operations on our API.

You can also learn the basics of our Inference Defend product.

RELATED INFORMATION

Updated at:

CalypsoAI © 2025-present