The idea: add one step to the pipeline that builds your API. It runs our open-source CI agent, which sends the service's OpenAPI spec to Graphwake with the build's context. On a pull request build the spec is stored as a draft, so the teams subscribed to a changed endpoint can hear about a breaking change before the merge. On a build of the live branch it becomes the live spec.

You need three things: the service's ingest key stored as a CI secret named GRAPHWAKE_KEY, the snippet for your platform, and the right live branch.

Example: what a consumer team sees once the step is in place.

Before you start

You need a service in Graphwake that takes its spec from CI. In Graphwake, add a service and choose CI pipeline under "Where the spec comes from". Graphwake shows the service's ingest key (dsk_...) only once, so copy it straight away.

You also need a build step that writes the spec. The agent sends the spec as it is, YAML or JSON. It doesn't generate it. The snippets below expect openapi.json in the working directory. Recipes for ASP.NET Core, Spring Boot, FastAPI, NestJS and Go are in the agent README.

Keys are per service. A repository with several services runs the step once per service, each with its own spec and its own key.

If you're on GitHub Actions or GitLab CI, the same agent has a ready-made action and template, described in inside the Graphwake CI agent.

Step 1: store the ingest key as a secret

Every snippet reads the key from the GRAPHWAKE_KEY environment variable. Keep it in your CI's secret store and pass it as that variable. The agent also accepts --key, but command-line arguments show up in process lists, so use the variable.

• Azure DevOps: add a pipeline variable named GRAPHWAKE_KEY and mark it secret. Azure doesn't hand secret variables to scripts on its own, which is why the snippet maps it in env.
• Bitbucket Pipelines: add a secured repository variable named GRAPHWAKE_KEY.
• Jenkins: add a credential of kind Secret text with the ID graphwake-key. The snippet binds it to GRAPHWAKE_KEY with withCredentials.

The agent hides the key in everything it prints. If the key leaks or is lost, rotate it from the service's Source tab in Graphwake and update the secret. The old key keeps working for 24 hours, which gives you time to update every pipeline.

Step 2: add the step

Azure DevOps

Add this script step to the pipeline's steps, after the step that writes openapi.json. It downloads the agent release, checks it against checksums.txt and runs it. This is azure-pipelines.yml, from the agent README:

  - bash: |
      VERSION=0.1.0
      BASE=https://github.com/graphwake/agent/releases/download/v$VERSION
      ARCHIVE=graphwake_${VERSION}_linux_amd64.tar.gz
      curl -fsSLO "$BASE/$ARCHIVE" && curl -fsSLO "$BASE/checksums.txt" &&
        sha256sum --check --ignore-missing checksums.txt &&
        tar -xzf "$ARCHIVE" graphwake &&
        ./graphwake upload --spec openapi.json || echo "graphwake: agent not run, spec not sent"
    displayName: Upload spec to Graphwake
    continueOnError: true
    env:
      GRAPHWAKE_KEY: $(GRAPHWAKE_KEY)

Run it on pull requests and on main. The agent treats a build with BUILD_REASON=PullRequest as a pull request build. For Azure Repos, pull request builds come from a build validation branch policy on the target branch; for a GitHub repository, from the pipeline's pr trigger.

If your organisation has the Graphwake extension installed, the GraphwakeUpload@0 task does the same with two inputs, spec: openapi.json and key: $(GRAPHWAKE_KEY).

One Azure quirk: it doesn't report a pull request closed without merging. Drafts of a pull request that never merges close on their own after 14 days.

Bitbucket Pipelines

Define the step once with an anchor and run it for every pull request and for builds of main. Replace the first script line with your own spec build, or put the step after the one that produces openapi.json. The step's image needs curl, sha256sum and tar. This is bitbucket-pipelines.yml, from the agent README:

definitions:
  steps:
    - step: &graphwake
        name: API spec
        script:
          - ./build-openapi.sh   # produce openapi.json
          - >-
            VERSION=0.1.0; BASE=https://github.com/graphwake/agent/releases/download/v$VERSION;
            ARCHIVE=graphwake_${VERSION}_linux_amd64.tar.gz;
            (curl -fsSLO "$BASE/$ARCHIVE" && curl -fsSLO "$BASE/checksums.txt" &&
            sha256sum --check --ignore-missing checksums.txt && tar -xzf "$ARCHIVE" graphwake &&
            ./graphwake upload --spec openapi.json) || echo "graphwake: agent not run, spec not sent"
pipelines:
  pull-requests:
    '**':
      - step: *graphwake
  branches:
    main:
      - step: *graphwake

The agent reads BITBUCKET_PR_ID on pull request builds and BITBUCKET_BRANCH on branch builds. If your live branch isn't main or master, rename the branches entry and set the live branch (step 3).

Jenkins

Use a multibranch pipeline, so Jenkins sets CHANGE_ID, CHANGE_URL and CHANGE_AUTHOR on pull request builds and BRANCH_NAME on branch builds. That's how the agent tells them apart, so a plain pipeline job won't do. Add this stage to the Jenkinsfile after the stage that writes openapi.json. The agent runs from its image, so the Jenkins agent needs Docker. The -e flags pass the build's variables into the container. From the agent README:

stage('API spec') {
  steps {
    withCredentials([string(credentialsId: 'graphwake-key', variable: 'GRAPHWAKE_KEY')]) {
      sh 'docker run --rm -e GRAPHWAKE_KEY -e BRANCH_NAME -e CHANGE_ID -e CHANGE_URL -e CHANGE_AUTHOR -e GIT_COMMIT -e JENKINS_URL -v "$PWD:/w" -w /w ghcr.io/graphwake/agent:0.1.0 upload --spec openapi.json || true'
    }
  }
}

The image has no git. On Jenkins that means the agent can't check that the build is still the branch tip, and it can't find the merged pull request, so a merge shows up in Graphwake as a separate live change. To get both, download the binary in an sh step instead, with the checksum-checked script from the README's Install section; it then uses the git in your workspace. The same script works if the Jenkins agent has no Docker. If you keep the image, pin it by digest (ghcr.io/graphwake/agent@sha256:...).

Step 3: set the live branch

A build of a live branch publishes the live spec, a pull request build stores a draft, and any other build sends nothing. None of these three platforms tells the agent its default branch, so the agent treats main and master as live unless you configure something else.

Only pull request builds and live-branch builds send anything.

The value is a comma-separated list and globs are allowed: main,release/* publishes from main and from every release/ branch. Before a live send the agent checks that the build is still the tip of its branch, where it can, so an old re-run doesn't replace a newer live spec.

Step 4: confirm it worked

Run the pipeline on a pull request that touches the API, then on main. The step prints one line, which tells you what happened:

Then open the service in Graphwake. Its Source tab shows the CI platform the agent detected and the time of the last upload.

To see an alert end to end, turn on Alerts on open pull requests under Settings → Alerts, subscribe a test team to an endpoint, and open a pull request that removes a response field from it. The team hears about it in its Slack, Microsoft Teams or email channel, and the impact score shows up on the change in Graphwake.

From there it works the same as on GitHub or GitLab: only the teams subscribed to a changed endpoint are told, nothing goes to a company-wide channel, and the step sends the OpenAPI file and the build's context. Your source code is not sent. Each change set gets an impact score with public rules.

Originally published on graphwake.com.

Which CI does your API build on? If it's not GitHub Actions, what's the integration you miss most?

It reads your code, not the other team's

Ask your AI assistant to remove a field from a response and it will do a careful job. It searches the repository, finds no reader of that field, updates the spec, fixes the tests and writes a tidy commit message. From inside the repository, that's the right answer.

The mobile app that reads the field lives in another repository, owned by another team, and nobody told the assistant it exists. The change is correct for the code it could see and wrong for the code it couldn't.

What the assistant sees, and what it doesn't

A smarter model doesn't help here: it reasons better about the same files, but it still doesn't see more of them.

Why it guesses instead of saying "I don't know"

When the answer isn't in what it can read, an assistant fills the gap with what's likely. "No usages found" becomes "unused". "Looks internal" becomes "safe to change". That's fine for a private helper function. For an API it's exactly the assumption that breaks another team in production.

A better prompt won't fix that. The assistant needs a source that actually knows who the consumers are.

Where the map can come from

To be fair, sometimes your assistant alone is enough. In a monorepo where every caller lives in the same repository, a code search finds them. If your consumers publish Pact contracts, the Pact Broker shows who depends on you. And if every caller identifies itself at the gateway and you can tie each client to a team, recent traffic is a good list.

Give the assistant the map, not more code

That's what we built Graphwake to keep. The map comes from your OpenAPI specs and from subscriptions: each consumer team declares the endpoints it calls, and owners are named per service or endpoint. Graphwake never reads code. It needs only the spec.

With Graphwake MCP, your AI assistant asks that map directly. Graphwake has no language model of its own. The model is your assistant, and Graphwake answers its questions with data. It works with Claude Code, Cursor, GitHub Copilot and other clients that accept an API key.

Example: two questions your assistant can now answer with data instead of a guess.

What it can ask:

• "Who consumes this?" returns the owner and the consumer teams of an endpoint, with the channel each one is alerted in.
• "What would this break?" runs a dry run of your local spec against the live one. Nothing is stored and nobody is alerted.
• "Who might we be missing?" brings in Discovery, which suggests teams that may use an endpoint, with the reason for each. Those suggestions come from fixed rules, not a model.

A local check sends only the OpenAPI file, which is compared and then deleted. Graphwake never sees your code.

Questions to ask before you change an endpoint

• Who owns this endpoint, and who consumes it?
• Which of those teams would this change break, and how badly?
• Is there an endpoint here that nobody has claimed?
• Is another open pull request already breaking the same endpoint?

Your assistant can answer what changes from your repository. Who it reaches, and who owns it, needs the map. The setup guide connects it in a few minutes.

Originally published on graphwake.com.

Has your assistant ever told you a field or endpoint was unused when another team was relying on it? How did you find out?

We built Graphwake around a smaller ask: the OpenAPI spec you already publish, plus one subscription from each team that calls an endpoint. That catches the common breaking changes on every endpoint. It doesn't replace tests where behaviour matters, and we'll come back to where those still pay off.

Why adoption stalls

Pact describes itself as "a code-first tool for testing HTTP and message integrations using contract tests", and its contract "is generated during the execution of the automated consumer tests" (docs.pact.io, as of 2026-11-14). That's where the cost comes from, and the people who pay it aren't the people who change the API.

The consumer pays. A team that calls ten internal APIs is asked to write and keep contract tests for ten services it doesn't own, and each test only pays off when some other team changes something. That's hard to get into a sprint.

Ownership is split. The consumer writes the contract, the provider verifies it, and someone runs the broker. When a verification fails at 6 pm, it isn't obvious whose job it is. Tests nobody owns rot, and a rotting contract test does real harm because it teaches people to ignore red.

Coordination costs the most. Every pair of teams has to agree on provider states, versions and the order of rollout. Pact's own page on what it is good for lists "testing providers where the consumer and provider teams do not have good communication channels" as a poor fit. In a company with dozens of services, that can describe a fair share of the pairs.

And coverage stops at the first team that says no. The same page lists an API whose other side "will not also be using Pact" as a poor fit. If the mobile team or a contractor team has no tests, the provider is back to guessing about the consumers it knows least about. Pact's docs are candid about it: "in the wrong situation, it will probably be just as painful!" A company-wide rollout plan tends to assume every pair of teams is in the right situation.

Spec-driven tools such as Specmatic generate the tests from the provider's spec instead (as of 2026-11-14). The contract is then the provider's description of the API, but the spec by itself still doesn't say which teams rely on which part of it.

What the spec alone can tell you

If you publish an OpenAPI spec and keep it accurate, much of the contract is already written down. Compare the spec before and after a change, and the changes that can make an existing client fail show up without anyone writing a test:

• an endpoint or a response field removed
• a required parameter or a required request field added
• a field's type changed
• the accepted values narrowed, for example a shorter enum

Example: two breaking changes found from the spec alone, with no test code.

That covers the shape of most breaking changes between internal teams. The check runs on every pull request that touches the spec, and the consuming teams don't have to do anything for it.

What the spec can't tell you

Behaviour, for one. A total that switches from dollars to cents is the same number in the spec. You only see that by running the service.

Drift. If the spec and the code disagree, a spec check is checking the wrong thing. Generating the spec from the code in the build helps, but it doesn't prove what the service returns.

What a consumer relies on beyond the spec: an undocumented field, the order of a list, an error message someone parses with a regular expression. None of that is in the spec.

And who calls the endpoint. A spec check can say "total was removed from GET /orders/{id}", but not which teams read total, or even which teams call GET /orders/{id}. So the finding goes to the pull request author, who usually doesn't know either.

The first three are reasons to keep contract tests where they earn their cost. The last one is why most breaking changes still surprise someone. The diff found the change, and the news went to the wrong person.

The smallest contract

A consumer-driven contract holds two kinds of information. Most of it is detail: the exact fields, values and states this consumer depends on. The rest is a single fact, that this team calls this endpoint. Only some integrations need the detail, which is expensive to write. Every integration needs the fact, and it costs almost nothing.

So in Graphwake we ask every consumer team for the fact and nothing more. A team that marks "we call GET /orders/{id}" has written the smallest contract there is: one subscription, with no test code and nothing to verify. Put that next to the spec diff and you get most of what teams wanted from contract testing. When the spec changes in a breaking way, the teams that call the endpoint hear about it, and with a CI step they hear while the change is still a pull request.

Example: the subscriptions behind an endpoint, and an endpoint nobody has claimed yet.

How that works in practice:

• Consumer teams subscribe to the endpoints they call. That's all we ask of them. They write no test code and run no broker. Subscriptions are per endpoint, not per field.
• Graphwake reads only your OpenAPI spec, from a URL it checks every 30 minutes or from one CI step on each pull request. It never asks for access to your code.
• A removed field, a new required parameter, a changed type or narrowed values reaches only the teams subscribed to that endpoint, in Slack, Microsoft Teams or email. Additions stay quiet by default.
• Each change set gets an impact score from 0 to 100 with public rules. A breaking change on an endpoint nobody subscribes to counts as a blind spot, and rule-based Discovery suggests teams that may call it.

The spec plus a list of who calls what also covers every endpoint from the first day, not only the ones whose consumers volunteered to write tests.

Where contract tests still earn their cost

Keep contract tests where both sides are teams that talk and will maintain them, where a runtime break would be costly (contract tests check the running service without deploying everything), and where consumers depend on behaviour or fields the spec doesn't describe.

The split we suggest: the spec and subscriptions on every endpoint, so every breaking change reaches the teams that call it, and contract tests on the integrations where a runtime break would be costly. If you already run Pact or Specmatic, keep them. Graphwake reads the spec and tells the teams subscribed to a changed endpoint; contract tests check a running service. They work side by side.

Originally published on graphwake.com.

Did your team roll out contract testing across services? How far did it get before it stalled, or did it stick?

What you're really asking about is a relationship between teams and endpoints. So when we built Graphwake, we stored it as one, in a graph. It turns out to need very little: three kinds of node, three kinds of edge, and queries that are one or two hops long.

Three nodes, three edges

The consumer graph. An endpoint with no subscriber is an orphan.

Here's the same model drawn for one example service. orders-api is owned by payments-team. Its endpoint GET /orders/{id} has two subscribed consumer teams, checkout and mobile-app, and one team Discovery suggests, storefront. GET /orders/{id}/invoice has no subscriber at all: an orphan, shown as a blind spot. The rest of this post uses that example.

A sketch of the model

The graph is small and shallow, so four relational tables hold it comfortably. This is a generic sketch of the shape, not Graphwake's schema:

-- Example: a minimal relational sketch of the model. Not Graphwake's schema.
CREATE TABLE teams (
  id    uuid PRIMARY KEY,
  name  text NOT NULL UNIQUE
);

CREATE TABLE services (
  id             uuid PRIMARY KEY,
  name           text NOT NULL UNIQUE,
  owner_team_id  uuid REFERENCES teams (id)          -- optional: owns
);

CREATE TABLE endpoints (
  id             uuid PRIMARY KEY,
  service_id     uuid NOT NULL REFERENCES services (id),  -- exposes
  method         text NOT NULL,
  path           text NOT NULL,
  owner_team_id  uuid REFERENCES teams (id),         -- optional override
  removed_at     timestamptz,                        -- kept for history
  UNIQUE (service_id, method, path)
);

CREATE TABLE subscriptions (                         -- subscribes to
  team_id      uuid NOT NULL REFERENCES teams (id),
  endpoint_id  uuid NOT NULL REFERENCES endpoints (id),
  created_at   timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (team_id, endpoint_id)
);

A few choices in there are worth spelling out.

Ownership is a nullable column rather than a table. One team answers for a service, and an endpoint may name a different one. It may stay empty, because an unowned endpoint you can see is more useful than a wrong name.

A subscription is a row with a composite key, so a team subscribes to an endpoint once. There's no field column, because a subscription covers the whole endpoint.

Endpoints aren't deleted. When an endpoint disappears from the spec it's marked removed. The live graph hides it, and the change history still points at it.

Edges have history too. Graphwake records each subscribe and unsubscribe, each ownership change, and each published change with the teams it reached. So it can later show that checkout heard about a change because it subscribed to GET /orders/{id} three months earlier.

Why the consumers keep the list

The producer team looks like the natural owner of a consumer list, but it's in a poor position to keep one. The producer sees requests arrive, but not which team's code sent them or what that code relies on.

Each consumer team knows exactly which endpoints it calls, and it learns this at a precise moment: when it writes the integration. Recording the edge at that moment takes a minute. Reconstructing it later from the producer's side takes a search across every repository, and that search still misses some callers.

So a subscription is a consumer team saying "we call this". The side that has the knowledge declares the edge, and nobody has to infer it.

The list stays current because we route every breaking change along it. A team that missed an alert finds out why and subscribes. A team that keeps getting alerts it doesn't need removes the edge.

A team that isn't subscribed isn't told, and the producer can see that this was a decision rather than an oversight. In a broadcast channel there's no way to tell a team that isn't affected from one that missed the post. And an endpoint with no subscriber shows up as an orphan, with a count next to it, so the gap is on screen rather than in someone's head.

Traffic logs and code search still help you find callers you don't know about yet (API consumer tracking compares those methods). They tell you which client called recently, or which repository contains a URL. A subscription tells you which team wants to hear when the contract changes, and that's what you need to know when a change breaks it.

Three questions the graph answers

Who does this change affect?

Start from the endpoints a change set touches and follow the subscription edges back to teams. One hop.

-- Example: which teams does a change to these endpoints reach?
SELECT DISTINCT t.name
FROM subscriptions s
JOIN teams t ON t.id = s.team_id
WHERE s.endpoint_id = ANY ($1);   -- the endpoints with a breaking change

In the example, a breaking change to GET /orders/{id} reaches checkout and mobile-app, and no other team. A breaking change to GET /orders/{id}/invoice reaches nobody, and that's the result to worry about. Graphwake counts it as a blind spot in the change's impact score, so the gap shows while the change is still a pull request.

What does my team depend on?

Start from a team, follow its subscription edges to endpoints, then each endpoint's owner edge (or its service's) to the team to ask. Two hops.

-- Example: what does one team depend on, and whom does it ask about each endpoint?
SELECT sv.name AS service, e.method, e.path,
       ot.name AS owner            -- the endpoint's owner, else the service's
FROM subscriptions s
JOIN endpoints e   ON e.id = s.endpoint_id AND e.removed_at IS NULL
JOIN services sv   ON sv.id = e.service_id
LEFT JOIN teams ot ON ot.id = COALESCE(e.owner_team_id, sv.owner_team_id)
WHERE s.team_id = $1
ORDER BY sv.name, e.path, e.method;

For checkout, that's GET /orders/{id} in orders-api, owned by payments-team, plus whatever else checkout calls. A team wants this list in front of it when it plans an upgrade or runs an incident review. Your AI assistant can read the same list through Graphwake MCP.

Which endpoints have no owner or no consumer?

Look for endpoints with a missing edge: no subscription, or no owner on either the endpoint or its service.

-- Example: live endpoints with no consumer (orphans) or no owner
SELECT sv.name AS service, e.method, e.path,
       NOT EXISTS (SELECT 1 FROM subscriptions s WHERE s.endpoint_id = e.id) AS orphan,
       COALESCE(e.owner_team_id, sv.owner_team_id) IS NULL                  AS unowned
FROM endpoints e
JOIN services sv ON sv.id = e.service_id
WHERE e.removed_at IS NULL
  AND (NOT EXISTS (SELECT 1 FROM subscriptions s WHERE s.endpoint_id = e.id)
       OR COALESCE(e.owner_team_id, sv.owner_team_id) IS NULL);

An orphan is either unused or unmapped, and you can't tell which from the graph alone. An unowned endpoint has nobody to ask. Graphwake flags orphans on the service graph and, when the weekly report is turned on, emails admins a weekly list of orphan endpoints, endpoints without an owner and services without an owner.

Do you need a graph database for any of this? Usually not. The useful questions are one or two hops, and a few relational tables with the right keys answer them. A graph database starts to pay off when you need long, variable-length paths, which a team-to-endpoint map rarely does.

Filling the gaps: Discovery

A declared graph is only as complete as its declarations. New endpoints and forgotten subscriptions need a nudge.

Discovery is rule-based. After every spec arrives, it looks at each new endpoint and each endpoint with a breaking change, and asks which teams that aren't subscribed may be affected. It uses only data the graph already holds, and it shows every signal it found as the reason for the suggestion.

Each signal adds a fixed weight and a past "Not us" subtracts one, with the total kept between 0 and 100. There's no language model involved. Suggestions under 40 are counted but not listed, so weak guesses stay out of people's way.

A person always decides. The suggested team (or an admin) chooses Confirm, which subscribes the team, or Not us. A decision is final for that team and endpoint, and a subscribed team is never suggested. When a released change breaks an endpoint and a suggestion for it scores 60 or more, the suggested team gets one "This may affect you" note with Confirm and Not us links. A workspace can turn this off.

One weak signal isn't enough. storefront is listed because two signals agree: a nearby subscription and a shared schema. If the orphan GET /orders/{id}/invoice changed in a breaking way, the path rule would point at checkout and mobile-app, because both subscribe to GET /orders/{id} just above it. On its own that rule scores under 40, so those suggestions are counted but not shown, and the orphan stays a visible blind spot until a team subscribes.

What the model leaves out

Each of these omissions is deliberate, and each one keeps the graph small enough for people to maintain.

There are no field-level subscriptions. A team subscribes to a whole endpoint, so a removed field reaches every subscriber of that endpoint, including a team that never read it. The cost is an occasional alert a team marks as handled. In return, the list stays small enough for a person to keep accurate.

There's no traffic or code analysis. Graphwake reads only the OpenAPI spec. It doesn't read source code or request logs, so an edge exists because a team declared it or confirmed a suggestion.

Consumers are teams, not services. A breaking change is a conversation between people, so the graph records who to talk to, not which process opens the connection. A team has members and a channel; a service doesn't read Slack.

Each node has one owner. When two teams share ownership, questions tend to go unanswered. The model allows one team per service and one override per endpoint.

Follows aren't edges. Anyone can follow an endpoint and get its alerts by email, but a follow isn't a subscription: it never appears in the graph, the consumer list, the impact score or Discovery.

Originally published on graphwake.com.

Where does "who calls this endpoint" live at your company today? A catalog, a spreadsheet, gateway logs, or someone's memory?

Getting it right comes down to four habits. Force the policies and give each one both USING and WITH CHECK. Set the tenant every time a connection is handed out, before the first query. Add the policy in the same migration as the table. Write tests that try to read across tenants.

All the code below is generic example code with made-up names (app.current_tenant, projects, tasks), written for this post.

Why we care about this

Graphwake maps which teams consume each endpoint of a company's OpenAPI specs and alerts only the subscribed teams when a breaking change is coming. To do that it holds things a company treats as internal: the list of its services and endpoints, which team calls which API, who owns what, every spec version, the history of API changes and the channels its alerts go to.

Example: a weekly ownership report built from one company's service map.

A service map describes, fairly completely, how a company's systems fit together. Showing one row of it to another customer would be a serious failure, whatever the row says. So we didn't want isolation to depend on every query in the codebase remembering a filter. It has to hold in the database, including for queries someone writes next year without ever reading this post.

Three ways to split tenants in PostgreSQL

There are three common layouts, and each is a reasonable choice for some products.

Database per tenant gives the strongest separation and the easiest per-customer restore, and it costs the most to run: migrations fan out across every database, and a pool per tenant doesn't scale to thousands of small tenants.

Schema per tenant sits in the middle, but its isolation still depends on the application setting the right search_path. That's the same per-request discipline RLS asks for, without a policy to catch the mistake.

Shared tables with RLS are the cheapest to run and migrate, and in this layout alone the database itself catches a forgotten filter. In exchange, the policies have to be right on every table, every query pays for an extra predicate, and one large tenant shares indexes with everyone else.

For a product with many tenants of similar size and one schema for all of them, shared tables with RLS are usually the right default. That's what we chose: Graphwake runs on one PostgreSQL database with row-level security per tenant.

How a request reaches only its own rows

The application authenticates the caller and works out which tenant it belongs to. Before the first query, it puts that tenant on the database connection as a setting. Every tenant table has a policy that reads the setting, and PostgreSQL adds the policy to every query against the table. Application code keeps writing ordinary queries, and the database makes sure they only touch the current tenant's rows.

The shape of a policy

Every table that holds tenant data gets a tenant_id column, an index that starts with it, and a policy. The example uses a custom setting called app.current_tenant; any name with a dot in it works as a custom setting.

-- Example: a tenant-owned table and its policy
CREATE FUNCTION current_tenant_id() RETURNS uuid
  LANGUAGE sql STABLE
  AS $$ SELECT NULLIF(current_setting('app.current_tenant', true), '')::uuid $$;

CREATE TABLE projects (
  id         uuid PRIMARY KEY,
  tenant_id  uuid NOT NULL REFERENCES tenants (id),
  name       text NOT NULL,
  UNIQUE (tenant_id, id)
);
CREATE INDEX projects_tenant_idx ON projects (tenant_id);

ALTER TABLE projects ENABLE ROW LEVEL SECURITY;
ALTER TABLE projects FORCE ROW LEVEL SECURITY;

CREATE POLICY projects_by_tenant ON projects
  USING      (tenant_id = current_tenant_id())   -- which rows a query can see
  WITH CHECK (tenant_id = current_tenant_id());  -- which rows a write may produce

Going through it piece by piece:

USING decides which existing rows a SELECT, UPDATE or DELETE can see. Another tenant's row doesn't raise an error. The query just doesn't see it.

WITH CHECK decides which rows an INSERT or UPDATE may produce. When it's left out, PostgreSQL uses the USING expression for writes as well. We still write it out. It keeps the write rule visible next to the read rule, you need it anyway once the two differ, and it makes plain that an UPDATE can't move a row to another tenant by changing its tenant_id.

FORCE ROW LEVEL SECURITY makes the policy apply to the table's owner too. By default the owner is exempt, and in many setups the application connects as the role that created the tables. Superusers are never subject to policies, so the application shouldn't connect as one.

current_setting(…, true) returns NULL instead of raising an error when the setting is missing, and NULLIF(…, '') turns an emptied setting into NULL as well. tenant_id = NULL is never true, so a connection with no tenant sees no rows and can't write any. The policy fails closed.

Wrapping the lookup in a STABLE function keeps the expression in one place and lets the planner evaluate it once per query, so the comparison can use the tenant_id index.

Foreign keys need one more step. PostgreSQL checks referential integrity without applying the policies, so a plain project_id reference would accept the id of another tenant's project. Making the parent unique on (tenant_id, id) and referencing both columns keeps every link inside one tenant:

-- Example: a child row cannot point at another tenant's parent
CREATE TABLE tasks (
  id          uuid PRIMARY KEY,
  tenant_id   uuid NOT NULL,
  project_id  uuid NOT NULL,
  title       text NOT NULL,
  FOREIGN KEY (tenant_id, project_id) REFERENCES projects (tenant_id, id)
);

Carrying the tenant on a pooled connection

The setting lives on a database connection, and connections are pooled. The connection your request uses served another tenant's request a moment ago and will serve a third one next. So the rule is: set the tenant every time a connection is handed out, before the first query, and never trust a value a previous borrower left behind. A request with no tenant should set an empty value rather than inherit one.

There's more than one correct way to do that. The example below sets the tenant at the start of each transaction with set_config(name, value, true), the parameter-friendly form of SET LOCAL, which lasts until the transaction commits or rolls back. Setting it whenever the application takes a connection from the pool, before any query, follows the same rule. Either way, the current request always writes the value.

-- Example: what the application sends for one request
BEGIN;
SELECT set_config('app.current_tenant', $1, true);  -- $1 is the tenant id, sent as a bind parameter
                                                     -- true = local to this transaction
SELECT id, name FROM projects ORDER BY name;         -- no tenant filter needed: the policy adds it
UPDATE projects SET name = $2 WHERE id = $3;         -- another tenant's row is simply not found
COMMIT;                                              -- the local setting ends with the transaction

A few practical points:

• Use set_config with a bind parameter. SET LOCAL doesn't accept parameters, so it tempts people into building the statement with string concatenation, while set_config takes the tenant id as an ordinary parameter.
• Match the setting to its scope. A transaction-local setting only covers the transaction it was set in, so with that form, run tenant work inside a transaction. Whatever the scope, the fail-closed policy above means a query without a tenant returns no rows rather than wrong rows.
• Put it in one place. Set the tenant in the data-access layer, from the authenticated request, not in each handler. In .NET that's a connection or transaction interceptor; most ORMs and frameworks have an equivalent hook.

Adding RLS in migrations

Row-level security belongs in the same migration that creates the table. A table that exists for one deploy without its policy can be read across tenants by any query, and "we'll add the policy later" is how that happens. A small migration helper that emits the ENABLE, FORCE and CREATE POLICY statements for a table name makes the safe version the easiest one to write, and gives every table the same policy shape.

Bringing an existing table under RLS takes a few more steps: add the column, backfill it from a parent row, make it NOT NULL, index it, then enable and force the policy.

-- Example: bringing an existing table under RLS, in one migration
ALTER TABLE invoices ADD COLUMN tenant_id uuid REFERENCES tenants (id);

UPDATE invoices i SET tenant_id = c.tenant_id
  FROM customers c WHERE c.id = i.customer_id;      -- backfill from the parent row

ALTER TABLE invoices ALTER COLUMN tenant_id SET NOT NULL;
CREATE INDEX invoices_tenant_idx ON invoices (tenant_id);

ALTER TABLE invoices ENABLE ROW LEVEL SECURITY;
ALTER TABLE invoices FORCE ROW LEVEL SECURITY;
CREATE POLICY invoices_by_tenant ON invoices
  USING (tenant_id = current_tenant_id())
  WITH CHECK (tenant_id = current_tenant_id());

Each migration should also have a down step that removes the policy and turns RLS off in the reverse order. On a large table, run the backfill in batches before the migration that adds the constraint, so the ALTER TABLE statements stay short.

Testing isolation

Policies are code. The bugs that hurt are the ones where everything looks fine to the tenant you're testing with, so the tests have to look from the other side. Three kinds of test cover most of it.

The first is a catalog check: one query that lists every table with a tenant_id column that isn't under forced row-level security with a policy. Run it in CI after migrations and it catches the new table someone added without its policy.

# Example: fail the build if any table with a tenant_id column is not under forced RLS
out=$(psql "$TEST_DATABASE_URL" -At -v ON_ERROR_STOP=1 <<'SQL'
SELECT c.relname
FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
JOIN pg_attribute a ON a.attrelid = c.oid AND a.attname = 'tenant_id' AND NOT a.attisdropped
WHERE n.nspname = 'public' AND c.relkind = 'r'
  AND NOT (c.relrowsecurity AND c.relforcerowsecurity
           AND EXISTS (SELECT 1 FROM pg_policy p WHERE p.polrelid = c.oid));
SQL
) || exit 1
# Any table name printed here is a table some tenant can read across.
[ -z "$out" ] || { echo "$out"; exit 1; }

The second is a two-tenant test. Seed two tenants with rows in every table. With tenant 1 set, check that each table returns only tenant 1's rows, that asking for tenant 2's rows by id returns nothing, that updates aimed at tenant 2 change nothing, and that inserting a row for tenant 2 is rejected. Then check that with no tenant set, nothing comes back.

-- Example: run as the same database role the application uses, with two seeded tenants
BEGIN;
SELECT set_config('app.current_tenant', '11111111-1111-1111-1111-111111111111', true);

SELECT count(*) FROM projects;                                   -- expect tenant 1's rows only
SELECT count(*) FROM projects
  WHERE tenant_id = '22222222-2222-2222-2222-222222222222';      -- expect 0, even when asked for directly
UPDATE projects SET name = 'x'
  WHERE tenant_id = '22222222-2222-2222-2222-222222222222';      -- expect UPDATE 0
INSERT INTO projects (id, tenant_id, name)
  VALUES (gen_random_uuid(), '22222222-2222-2222-2222-222222222222', 'x');
                                                                 -- expect: new row violates row-level security policy
ROLLBACK;

BEGIN;                                                           -- no tenant set at all
SELECT count(*) FROM projects;                                   -- expect 0
ROLLBACK;

The third is a pool test. Run requests for tenant 1 and tenant 2 alternately through a pool with a single connection, and assert that each request only sees its own tenant. This one catches a setting that outlives its request.

Run all of them as the same database role the application uses, not as a superuser. A superuser isn't subject to policies, so an isolation test run as one passes whether the policies work or not.

What RLS doesn't do for you

• It doesn't choose the tenant. The policy trusts the setting. Resolving the tenant from the authenticated caller, and never from a value the caller can choose freely, is still the application's job.
• It doesn't replace your filters. Writing tenant_id into application queries helps the planner and makes the intent obvious. RLS is the backstop for the query that forgot.
• It doesn't isolate load. One heavy tenant still shares tables, indexes and the server with the others. Watch per-tenant query cost and index on tenant_id first.
• It covers rows and nothing else. Files in object storage, caches and search indexes need their own tenant scoping.

On performance: a simple policy adds one predicate to each query, much like the WHERE tenant_id = … you'd write by hand. With an index that starts with tenant_id and a STABLE function or a direct current_setting call in the policy, the planner can use the index and the overhead is small. Check the plans of your heaviest queries with EXPLAIN after enabling it.

How we do it in Graphwake

Graphwake runs on one PostgreSQL database. Customer data lives in tables that carry the workspace each row belongs to, and a row-level security policy filters each read and checks each write against it.

The API resolves the workspace from the signed-in user and puts it on the database connection before the first query runs. A connection with no workspace set sees no rows.

Migrations enable and force row-level security on customer tables, with the same policy shape on each one, so the policy applies to every query the application runs. Tests seed two workspaces and check that each one sees only its own rows, that one can't write into the other, and that a connection with no workspace sees nothing.

That's how one company's service map, subscriptions, ownership and API change history stay out of every other company's view, while its consumer teams get alerts about the breaking changes that affect them.

A checklist

1. Give every tenant-owned table a tenant_id column, an index that starts with it, and composite foreign keys on (tenant_id, id).
2. Enable and force row-level security, with one policy that has both USING and WITH CHECK and fails closed when no tenant is set.
3. Connect as an ordinary role, never as a superuser.
4. Set the tenant every time a connection is handed out, before the first query, with a bind parameter, in one place in the data-access layer.
5. Add the policy in the migration that adds the table, and run a catalog check in CI that finds any table without one.
6. Test with two tenants, with no tenant, and through a pool of one connection, as the application's own role.

Originally published on graphwake.com.

Which layout did you go with for tenant isolation, and would you pick it again?

You can find both by comparing the endpoints in your OpenAPI specs with two lists, consumers and owners, and clear them with a short weekly review. Here's how we'd do it by hand, and then how Graphwake does it for you.

Two gaps, two questions

The two are independent. An endpoint can have an owner and no known consumers, which is the common case for a new endpoint. It can have five consumer teams and no owner, which is common after a reorganisation. Both are worth a look, and they need different fixes.

A note on the word. In API security writing, "orphaned", "zombie" and "shadow" APIs are endpoints that stay reachable while nobody maintains or owns them, and tools find them from traffic: Cloudflare API Shield labels a saved endpoint with no traffic for 32 days cf-risk-zombie. Here, and in Graphwake, the word means a gap in the record: an endpoint in the spec with no consumer team. An orphan in this sense may be called all day by a team that never said so.

Why it matters

When an endpoint with no recorded consumers changes, the producer has no one to warn. Either nobody calls it, or someone does and will find out in production.

Retirement stalls, too. Without a consumer list nobody can say whether the endpoint is safe to remove, so it stays forever. (We wrote separately about how to deprecate an endpoint safely.)

Ownership decays quietly. Teams split, merge and rename. An owner field that points at a team that no longer exists is worse than an empty one, because it looks filled.

And there's a security angle. The OWASP API Security Top 10 lists improper inventory management (API9:2023): hosts without a clear purpose, no retirement plan, missing or outdated documentation. An endpoint nobody owns is part of that problem.

How to find them from your specs

Your OpenAPI specs list every endpoint that exists. Start there, not from traffic, so the list is complete on one side. With a JSON spec and jq, list each operation as a method and a path:

jq -r '.paths | to_entries[] | .key as $p | .value | keys[]
  | select(test("^(get|put|post|delete|patch|head|options|trace)$"))
  | "\(ascii_upcase) \($p)"' openapi.json | sort > endpoints.txt

Then compare it with whatever records consumers today: a sheet, a wiki page, catalog entries. In this example it's a two-column file, METHOD /path,team. The lines that only appear in the spec are your orphans:

cut -d, -f1 consumers.csv | sort -u | comm -23 endpoints.txt -

For a made-up orders-api with three of its four endpoints on record, the output is:

GET /orders/{id}/invoice

Do the same with an owners list to find endpoints without an owner. If owners are set per service, an endpoint without its own owner inherits the service's. Only an endpoint whose service has no owner either is truly unowned.

What to do with each orphan

Finding one means you have something to check. Look at the endpoint's traffic, then pick one of three answers.

Don't delete an orphan just because it's an orphan. Only remove it once the traffic shows nobody calls it over a full business cycle and the owner agrees.

For an endpoint with no owner, someone just has to decide: whoever owns the service, or the team that changes the code most often, takes it. Write it down even if it's a guess you'll correct later.

A weekly routine

1. Get the list on Monday. Orphan endpoints, endpoints without an owner and services without an owner, sent to the people who can assign owners. Short lists get read, so cap each one.
2. Triage the new lines. Each new orphan gets one of the three answers above. Each unowned service or endpoint gets an owner, or a name and a date for the decision.
3. Ask, don't chase. Send one message per likely consumer team, naming the endpoint and the evidence. A team that says "not ours" has given you an answer too.
4. Watch the count, not the list. The number should fall and then stay low. A sudden rise usually means a new service came in with no consumers mapped yet.

How Graphwake reports orphans

In Graphwake an orphan endpoint is a live endpoint that no consumer team subscribes to. Graphwake reads every endpoint from your OpenAPI specs and the subscriptions teams make, so the comparison above runs on its own and the weekly list arrives without anyone building it.

Example: the Monday email to admins.

Every Monday from 08:00 UTC, admins get one email listing orphan endpoints, endpoints without an owner and services without an owner, up to 50 lines each. A week with nothing to report sends nothing, and admins can turn it off.

Ownership is optional and inherits. A service may have an owner team, and an endpoint may have its own. An endpoint without one inherits the service's owner, so only real gaps show up.

The service graph flags each orphan endpoint next to its owner, and the overview counts them. A breaking change on an orphan counts toward the impact score as a blind spot.

When an orphan is new or changes in a breaking way, Discovery suggests teams from rules such as subscriptions to neighbouring endpoints and shared schemas. Someone confirms before a team subscribes.

When a traffic tool or catalog is enough on its own

If you need to know whether an endpoint is called at all, gateway analytics, access logs or a traffic-based inventory such as Cloudflare API Shield answer that. If you're hunting endpoints that are live but missing from any spec, API discovery tools that compare traffic with your specs find these "shadow" endpoints. And if you want one owner field for every service, library and database, a software catalog like Backstage covers far more than APIs.

Originally published on graphwake.com.

How many endpoints in your specs have a consumer on record somewhere? Has anyone on your team ever run the numbers?

The behaviour is Graphwake's. The company, people and numbers are made up for the example.

The cast

Four teams are involved, though only one of them writes the change.

That map comes from subscriptions: each consumer team marked the endpoints it calls. Graphwake uses that list to tell checkout and mobile-app and to leave search alone, so nobody has to remember who calls what during a review.

Example: the map for GET /orders/{id}. It also shows storefront, a Discovery suggestion nobody has confirmed, and an endpoint with no subscriber.

The timeline

Example: pull request #42 in orders-api, from the spec change to the change history.

1. The spec changes in a pull request

John renames the response field total to totalAmount and adds a region query parameter so orders can be served from the nearest region. He makes region required, because the new code needs it. Lines starting with + were added and the line starting with - was removed:

# orders-api/openapi.yaml, pull request #42 (branch orders-region-param)
paths:
  /orders/{id}:
    get:
      parameters:
        - { name: id, in: path, required: true, schema: { type: string } }
+       - { name: region, in: query, required: true, schema: { type: string } }
      responses:
        "200":
          content:
            application/json:
              schema:
                properties:
                  id:          { type: string }
-                 total:       { type: number }
+                 totalAmount: { type: number }
                  currency:    { type: string }

As a code diff, this is a small pull request. As a contract it holds three changes, and two of them break callers that work today.

2. The CI step sends the spec as a draft

orders-api runs the Graphwake CI agent in its pipeline. On the pull request build, the agent sends the spec with the build's context: pull request #42, its link, the author and the commit. Graphwake stores it as a draft for that pull request. The live spec stays exactly as it was, because nothing has been merged.

This step is why everything below happens before the merge. Without it, Graphwake can still poll a published spec URL every 30 minutes, but then it only sees the change after it's deployed.

3. Graphwake finds the breaking changes

Graphwake compares the draft with the live spec, endpoint by endpoint. We call a change breaking when it can make an existing client fail, which in practice means one of these:

• an endpoint or a response field is removed
• a new required parameter or required request field is added
• a field's type changes
• the accepted values are narrowed, for example an enum loses a value

The rule of thumb behind the list: does a client that follows today's contract keep working? Adding something a client can ignore is safe. Taking away something it reads, or demanding something it doesn't send, breaks it. A team that wants to hear about the safe changes too can opt in to every change.

4. The impact score is computed

Each change set gets a score from 0 to 100, made of four parts with public rules. Each part counts something, gives a fixed number of points per item and stops at a cap, so no single part can dominate. The full rules are on the impact score page. For #42:

The two breaking changes are the removed field and the new required parameter; the added field doesn't count. checkout and mobile-app subscribe to GET /orders/{id}, search doesn't, so two affected teams. The people count is the distinct members of those teams: 4 in checkout and 6 in mobile-app, nobody in both. Someone who follows the endpoint personally gets the alert by email but isn't counted here. And there are no blind spots, because every endpoint with a breaking change has at least one subscriber and no endpoint was removed.

54 out of 100 is Medium. Low runs from 0 to 29, Medium from 30 to 59, and High is 60 and above. We publish the rules so that a reviewer can check every point.

Blind spots work differently. Suppose #42 had also removed GET /orders/{id}/invoice, which no team subscribes to. That's a third breaking change (30 instead of 20) and a blind spot (plus 10): 74 out of 100, High. The score goes up because Graphwake has nobody to tell. When that happens, Discovery suggests teams that may be affected, each with a reason, and a suggested team becomes a subscriber only when someone confirms it.

To show what High looks like, here's a larger example from the same service. This one is not #42: pull request #1284 has three breaking changes and three affected teams, so 30 + 36 + 13 people = 79.

5. Two teams are alerted, the third is not

With "Alerts on open pull requests" turned on in settings, Graphwake sends the alert while #42 is still open. It goes to the teams subscribed to the changed endpoint, each in the channel that team chose: checkout in #team-checkout, mobile-app in #mobile-alerts. A team that chose Microsoft Teams or email gets the same information there.

These are the lines of the chat message checkout and mobile-app get. The Change line holds a short description of each breaking change found, separated by semicolons:

Breaking change in PR #42
Service: orders-api
Endpoint: GET /orders/{id}
Owned by payments-team
Change: <the removed field>; <the new required parameter>
Author: john
PR: https://github.com/example/orders-api/pull/42

The message carries what a consumer needs to act on it: the endpoint, what changed, who changed it and where to comment. The impact score stays in Graphwake, on the change set, next to the list of teams it reached.

search gets nothing. It calls orders-api, but not the endpoint that changed. In Graphwake's impact view it's listed among the teams that consume the service but none of the changed endpoints, so the producer can see it was considered and that skipping it was deliberate.

While the pull request stays open, Graphwake keeps one message per team and endpoint. If John pushes again, the existing message is edited in place instead of a new one being posted. An edit notifies nobody, so if a push makes a change breaking that wasn't before, Graphwake adds a thread reply: "This change is now breaking." If a push removes the breaking change, the message is updated to say it's no longer breaking.

6. The teams prepare, then the pull request merges

checkout starts sending region, mobile-app moves to totalAmount, and each team marks the change as handled for itself in Graphwake. That conversation happened during the review. Usually it happens after an incident.

John merges #42. The default-branch build sends the spec as live, and the agent reports that the commit came from pull request #42. Graphwake settles the draft as merged: each team's message is updated to "Published: Breaking change in PR #42", with the thread reply "Published to live." The handled marks carry over, because the breaking changes were the same in the draft and at the merge. Had John closed the pull request instead, the reply would have been "PR closed without merging." Where the CI doesn't report closes, the draft closes quietly after 14 days, and the live spec never changed.

7. The change history keeps the record

The merge is recorded in the change history of orders-api and of GET /orders/{id}: the change set, its breaking changes, the team that published it, and which teams were told in which channel. The history is append-only. Months later, when someone asks why checkout knew about the region parameter, checkout's subscription to the endpoint links to the change it was told about.

If an old pipeline on the main branch is re-run later, its spec doesn't quietly undo #42: Graphwake ignores a live spec from an older commit than the one it already holds.

Why routing beats broadcasting

The usual alternative is a post in a shared channel, and the hope that everyone who calls the endpoint reads it. In this example that means asking the whole company to work out whether they call GET /orders/{id}.

With subscriptions, each consumer team records what it calls once, when it starts calling it. Nobody has to repeat that matching every time another team ships.

search heard nothing, and that tells search something: it isn't affected. In a broadcast channel, silence tells you nothing: you can't tell "not affected" from "missed the post".

And an endpoint nobody subscribes to shows up as a blind spot and raises the score, where a broadcast would leave it to surface in production.

What to take from this

1. Review spec changes as contract changes. A rename is a removal, and a new required input breaks every caller that doesn't send it.
2. Count who is affected before the merge. Two breaking changes on an endpoint with two consumer teams is a different conversation from the same changes on an endpoint nobody calls.
3. Tell the teams that call the endpoint, in their own channel, and let everyone else stay quiet.

For the review side, there's a breaking change checklist for pull requests, and more examples of what counts as breaking in OpenAPI breaking changes.

Originally published on graphwake.com.

When a breaking change shipped at your company, who found out first: the team that made it, or the team whose app broke?

The code is at github.com/graphwake/agent under Apache-2.0. Below are the parts that took us the most thought.

Why the agent exists

Graphwake keeps a map of which teams consume each endpoint of your OpenAPI specs. When a spec changes in a breaking way, it scores the change and alerts only the teams subscribed to the affected endpoints, in Slack, Microsoft Teams or email. For that to be useful, the alert has to arrive while the change can still be stopped or adjusted.

A spec can reach Graphwake two ways. Graphwake can poll a published spec URL every 30 minutes, which sees a change once it's deployed. Or a step in the producer's pipeline sends the spec from the pull request build, so Graphwake sees the change before anyone merges it. The agent is that step.

We couldn't assume one CI system. A company with a few dozen services often runs two or three. So we set ourselves three goals.

One step, same settings everywhere. --spec and GRAPHWAKE_KEY are enough on every platform. Everything else is read from the CI environment.

Only the spec. The agent sends the spec file as it is (YAML or JSON). It doesn't generate it, and your source code is not sent. To classify the build it reads the CI environment and, where git is available, a few commit details: the SHA, the commit time and message, and the remote branch tip.

Small enough to read. It's written in Go against the standard library only and ships as one static binary. You run it with a secret in your pipeline, so you should be able to read every line of it.

One binary, several ways to install it

Releases are built with GoReleaser: binaries for Linux, macOS and Windows on amd64 and arm64, a checksums.txt, and a multi-arch image ghcr.io/graphwake/agent (distroless, with a busybox shell so GitLab CI and Jenkins can run scripts in it). Each platform gets the wrapper it expects, and every wrapper runs the same binary.

GitHub Actions, from the agent README. The closed type lets the agent report pull requests closed without merging:

on:
  pull_request:
    types: [opened, synchronize, reopened, closed]
  push:
    branches: [main]

jobs:
  api-spec:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      # ... build and produce openapi.json
      - uses: graphwake/agent/github-action@v0.1.0
        with:
          spec: openapi.json
          key: ${{ secrets.GRAPHWAKE_KEY }}

GitLab CI. Add GRAPHWAKE_KEY as a masked CI/CD variable:

include:
  - remote: https://raw.githubusercontent.com/graphwake/agent/v0.1.0/gitlab/graphwake.gitlab-ci.yml

graphwake:
  extends: .graphwake-upload
  needs: [build]
  variables:
    GRAPHWAKE_SPEC: build/openapi.json

Where you install the binary yourself, verify it the same way the wrappers do. For the image, pin it by digest (ghcr.io/graphwake/agent@sha256:...) to get the same guarantee. This is adapted from the README's install script:

VERSION=0.1.0
BASE=https://github.com/graphwake/agent/releases/download/v$VERSION
ARCHIVE=graphwake_${VERSION}_linux_amd64.tar.gz
curl -fsSLO "$BASE/$ARCHIVE" && curl -fsSLO "$BASE/checksums.txt" &&
  sha256sum --check --ignore-missing checksums.txt &&
  tar -xzf "$ARCHIVE" graphwake &&
./graphwake upload --spec openapi.json

Step one: which CI is this?

Every CI system sets its own environment variables, and each has one that identifies it. The agent checks them in a fixed order and falls back to other when none matches. From internal/ci/detect.go:

switch {
case getenv("GITHUB_ACTIONS") == "true":
    return detectGitHub(getenv, readFile)
case getenv("GITLAB_CI") != "":
    return detectGitLab(getenv)
case strings.EqualFold(getenv("TF_BUILD"), "true"):
    return detectAzure(getenv)
case getenv("BITBUCKET_BUILD_NUMBER") != "" || getenv("BITBUCKET_COMMIT") != "":
    return detectBitbucket(getenv)
case getenv("JENKINS_URL") != "" || getenv("JENKINS_HOME") != "":
    return detectJenkins(getenv)
}
return Info{Platform: Other, Event: EventUnknown}

getenv and readFile are injected, so every platform is tested from fixture environments and event payloads.

Each detect* function then reads that platform's own variables for the pull request number, link, author, commit and branch. Most of that is a lookup table. Four cases took more care.

On a GitHub pull request, GITHUB_SHA points at GitHub's temporary merge commit, not at the commit the author pushed. We read the pull request's head SHA from the event payload instead, so Graphwake records the author's commit.

Only push, workflow_dispatch and schedule may publish the live spec on GitHub. Events such as issue_comment or workflow_run also run on the default branch's ref, but they often build a pull request's code, so we don't trust them with the live spec.

Azure DevOps has two pull request ids. SYSTEM_PULLREQUEST_PULLREQUESTNUMBER is the GitHub number for GitHub-hosted repositories and SYSTEM_PULLREQUEST_PULLREQUESTID is the Azure Repos id. The agent takes the first one that is set.

On a change request, Jenkins puts PR-5 in BRANCH_NAME. The real source branch is in CHANGE_BRANCH.

Step two: what kind of build is it?

Three outcomes from one step. Only pull request builds and live-branch builds send anything.

A live branch is the platform's default branch when the platform says what it is (GitHub's event payload, GitLab's CI_DEFAULT_BRANCH), else main and master. Azure DevOps has no default-branch variable, so it uses the fallback. Teams that release from more than one branch list them. On GitHub Actions that's the action's live-branch input, and the same branches go in the workflow's push trigger. Elsewhere it's GRAPHWAKE_LIVE_BRANCH=main,release/*. The patterns are globs.

With the spec, the agent sends the build's context: the platform, the commit SHA, the commit time (the committer date from git when git is there, else the platform's value) and the branch the commit is on. If the branch name is longer than 255 characters we drop it rather than reject the request, so the spec still arrives.

Here's an example request body for a pull request build on GitHub Actions (POST /api/ingest/specs, with the key in the Authorization header):

{
  "spec": "openapi: 3.0.3\ninfo: ...",
  "context": {
    "kind": "pullRequest",
    "prId": "42",
    "prUrl": "https://github.com/example/orders-api/pull/42",
    "author": "john",
    "state": "open",
    "platform": "github",
    "commitSha": "9f2c1e4b7a0d",
    "commitTime": "2026-11-10T10:05:00Z",
    "branch": "orders-region-param"
  }
}

An old re-run must not replace a newer live spec

Live builds have a race that pull request builds don't. Say John merges pull request #41 at 10:00 and pull request #42 at 10:05. The 10:05 pipeline finishes first and publishes the live spec. Then the 10:00 pipeline finishes, or someone re-runs it an hour later. If it sent its spec as live, the API would look as if #42's changes had been reverted.

So before it sends a live spec, the agent checks two things.

1. The checked-out commit is the build's commit. git rev-parse HEAD must match the commit the platform says it is building. If they differ, the live spec is not sent.
2. The build's commit is still the branch tip. On GitHub the agent asks the API with the job token (GET /repos/{repo}/commits/{branch}, which answers with the SHA). That call needs only contents: read. Elsewhere it runs git ls-remote origin refs/heads/<branch>. If the branch has moved on, nothing is sent and the newer build sends the live spec.

This is the one line printed when a build is behind:

graphwake: this build is not the tip of main (a newer build sends the live spec), nothing sent

We made the check best effort on purpose. If it can't run, for example because the image has no git or the job has no credentials, the spec is sent anyway. git runs with GIT_TERMINAL_PROMPT=0 and a 15-second timeout, so a missing credential can't hang the job waiting for a password. The GitLab template's image has no git, so the tip check is skipped there.

The server has its own check as well. Every send carries the commit SHA and commit time, so Graphwake can ignore a spec from an older commit than the live one.

Which pull request did this merge come from?

On a live build the agent also tries to find out whether the commit came from a merged pull request, because Graphwake handles the two cases differently. "Pull request #42 merged" settles the draft Graphwake already holds for #42. A plain live spec is a new change with no pull request behind it.

On GitHub the agent asks the API with the same job token: GET /repos/{repo}/commits/{sha}/pulls, keeping the pull request that has merged_at set and whose base is the branch being built. That works for merge commits, squash merges and rebase merges. This call needs pull-requests: read. If your workflow sets permissions:, list it there next to contents: read.

Everywhere else, or when the API doesn't answer, it reads the commit message and looks for each platform's default merge message. From internal/ci/mergemsg.go:

githubMerge  = `\AMerge pull request #(\d+)\b`
githubSquash = `\A[^\n]*\(#(\d+)\)[ \t]*(?:\r?\n|\z)`
gitlabMerge  = `See merge request [^\s!]*!(\d+)\b`
azureMerge   = `\AMerged PR (\d+):`
bitbucketPR  = `\(pull request #(\d+)\)`

Jenkins and unknown CIs try all of them, because they can sit in front of any git host.

That covers Merge pull request #12, a squash title ending in (#12), GitLab's See merge request group/project!12, Azure's Merged PR 12: and Bitbucket's (pull request #12). Teams that write custom merge messages still get the live spec. They just see the merge and the live change separately.

Keeping the ingest key out of logs

CI logs are widely readable and kept for a long time. The agent prints exactly one line per run, and that line goes through a single reporter that knows the secrets. From internal/report/report.go, where every value registered with Hide is replaced before anything is printed:

func (r *Reporter) clean(s string) string {
    for _, secret := range r.secrets {
        s = strings.ReplaceAll(s, secret, "***")
    }
    s = strings.Join(strings.Fields(s), " ") // one line, no control characters
    ...
}

A few more rules around that reporter:

• Secrets are registered before flags are parsed: GRAPHWAKE_KEY, the GitHub token, and any value given after --key. A typo in the flags can't echo the key back.
• Flag errors never show values. We discard the parser's own output. An unknown flag is reported as "unknown flag" without its name, because a key pasted as -dsk_... would otherwise end up in the log.
• The GitHub Action masks the key too (::add-mask::), even when it doesn't come from a secret.
• --key works, but command-line arguments show up in process lists. The README recommends GRAPHWAKE_KEY from a CI secret.
• Pull requests from forks get no secrets. The step prints a warning and sends nothing.

On Graphwake's side, an ingest key belongs to one service and is stored only as a SHA-256 hash. A repository with several services runs the step once per service, each with its own spec and key.

What happens after the spec arrives

1. Pull request build: a draft. The spec is stored as a draft for the pull request. Graphwake finds the breaking changes and scores the change set from 0 to 100 with public rules.
2. Consumer teams hear before the merge. With "Alerts on open pull requests" turned on in settings, the teams subscribed to the changed endpoints are told in Slack, Microsoft Teams or email. The alert names the endpoint, what changed, who changed it and which pull request did it.
3. Merge: the spec goes live. The default-branch build sends the spec as live. If the commit came from a pull request, that pull request's draft notifications are settled as "merged" and the real notifications go out.
4. Closed without merging: the draft goes away. Only GitHub reports a close. The agent sends it and the drafts are discarded. On the other platforms, drafts of a pull request that never merges close on their own after 14 days.

So the teams that call a changed endpoint hear about it while it's still a pull request, not after a deploy, and only those teams: consumer teams subscribe to the endpoints they use, and nothing goes to a company-wide channel. The setup is the same two settings, the spec and the key, on all five CI systems, and the agent sends the OpenAPI file and the build's context, nothing else.

Build it yourself

The logic above lives in one Go module with no dependencies outside the standard library. The wrappers are a few lines of shell, YAML and Node, and each platform's environment has its own tests. It runs inside your pipeline with one of your secrets, so you shouldn't have to take its behaviour on trust:

go vet ./... && go test ./...
go build -o graphwake ./cmd/graphwake
./graphwake version

On any CI we don't detect, pass --pr on pull request builds and --live on release builds.

Originally published on graphwake.com.

How many CI systems does your company run side by side? We went with environment sniffing plus a fallback flag; curious whether anyone has found a cleaner way to classify builds across platforms.

Here are the rules, in full.

The four parts

Each part scores the lower of two numbers: its count times its points, or its cap. The caps add up to 100.

The bands

A change set with no breaking change scores 0. Nothing breaks, so there's nothing to weigh.

Example: a different pull request (#1284) as it shows in Graphwake, each part with its arithmetic.

A worked example

Take a hypothetical change set that removes one response field and adds a required parameter on another endpoint.

• That's 2 breaking changes: 20 points.
• Three teams subscribe to the affected endpoints. 3 times 12 is 36, which is exactly the cap: 36 points.
• Together those teams have 9 people: 9 points.
• One of the two endpoints has no subscribers. That's 1 blind spot: 10 points.

20 + 36 + 9 + 10 = 75, which lands in High.

Why a blind spot counts as risk

A removed endpoint, or a breaking change on an endpoint with no subscribers, is a change whose reach is unknown. We could have scored it as zero, since nobody is subscribed and nobody gets alerted. That would reward exactly the situation you should worry about. Counting it pushes the team to find out who is affected before merging.

To help with that, Discovery suggests likely teams, with the reason for each suggestion.

What the score is for

It helps reviewers decide where to look first and who to talk to. Low means few or no teams are affected. High means several breaking changes, several teams, or blind spots, and the blind spots are where someone should go find out who calls those endpoints.

With the CI step, reviewers see the score while the change is still a pull request, before the merge.

Each score is shown with its four parts and the arithmetic, so "why did this get 75?" always has an answer on screen.

Originally published on graphwake.com.

Do you put any kind of risk number on API changes today? If you do, can your reviewers explain how it was computed?

Widely is the problem. An announcement is written for everyone, so it gets read as if it were for no one in particular. What you actually need is to reach the few teams whose code breaks, and a post written for everyone is a poor way to find them.

Why the announcement channel feels right

It's cheap: one post, no list to keep, and you don't need to know who your consumers are. It's visible, since anyone can scroll back and see what changed. And it covers you. After the incident you can point at the post and say "we announced it."

That last one is the honest reason. An announcement channel protects the sender. It does very little for the reader.

Why it stops working

Here's the shape of it, with made-up numbers (an example, not a measurement):

12 teams post in #api-changes
2 posts each a week  → 24 posts
you call endpoints of 2 teams
some of their posts touch them

→ 1 or 2 of 24 posts are yours

When 22 of 24 posts aren't about you, ignoring the channel is the rational choice. People skim, then they mute. The one post that was about you goes with the rest.

Every post also asks every reader to do the matching: do we call this endpoint? Most people can't answer that from memory, so they guess.

Often the team that needed the post isn't even in the channel. A new team, a team in another office, the contractor team that built the mobile app. Nobody added them.

Posted isn't received, either. The sender can't tell who read it, so "we announced it" and "they knew" turn into the same sentence in the postmortem.

Mentions make all of this worse. @here and @channel wake everyone up to tell a few. Slack itself asks for confirmation before one goes to a channel of six or more people, and lets owners restrict who may use them.

Telling is the sender's job

An announcement moves the work to the reader, who has to read everything to find what's theirs. A targeted message keeps the work with the sender, who knows what changed, and with a list of who calls each endpoint, which nobody can keep in their head.

That list is the hard part, and it's why the shared channel survives. If you don't know who calls GET /orders/{id}, posting to everyone is the only way to reach them. Swapping in a better channel doesn't change that; you need the list of who calls what.

What to do instead

What that message should say, and how to wire it into Slack and Microsoft Teams, is covered in API change notifications in Slack and Microsoft Teams.

How we built Graphwake around this

Graphwake is built on this opinion. Consumer teams subscribe to the endpoints they call, and a breaking change in your OpenAPI spec reaches only those teams, in the Slack, Microsoft Teams or email channel each one picked. Everyone else hears nothing, on purpose, apart from a one-time note to a team Discovery strongly suspects is a caller.

Example: one breaking change, three subscribed teams told, one team left alone.

A few details that matter in practice:

• The subscriber list stays current because the readers keep it. Each consumer team marks the endpoints it calls.
• Each team picks one Slack or Microsoft Teams channel, or email. We never add @here or @channel.
• With the CI step, the alert goes out while the change is still a pull request, and the same message is updated when it merges or closes.
• An endpoint nobody subscribes to counts as a blind spot in the impact score. Discovery suggests teams that may use it, and a team suggested at 60% or more gets a one-time "This may affect you" note when a breaking change goes live.

When a shared channel alone is enough

To be fair to the channel: in a small company where every engineer touches every service and there are a few changes a month, one channel already is the targeted channel. As a record, a feed of every change that nobody is expected to act on is a useful log. And for public APIs, customers outside your company aren't in your Slack anyway; a changelog and an email list reach them.

The trouble starts when the channel becomes the way consumers are supposed to find out about breaking changes.

Originally published on graphwake.com.

Does your company have an #api-changes channel? Do people actually read it, or has it quietly become a log?

This post walks through the setup line by line, shows what the output looks like on a real change, how to tune it, and, the part most guides skip, what to actually do with each finding.

What you need

• A GitHub repository with an OpenAPI 3 spec committed to it. The examples use openapi.yaml in the repo root; change the path to yours.
• If your spec is generated from code, a build step that writes it to a file first. The action compares files, not code.
• That's it. The free actions run the oasdiff CLI inside your runner, with no account and no token.

The oasdiff/oasdiff-action repository holds several actions, one per folder:

This post uses breaking. Note that @v0 is a moving tag that follows every stable v0.x.y release; pin a release tag if you want to choose when to upgrade.

Step 1: add the workflow

Create .github/workflows/oasdiff.yaml:

name: oasdiff
on:
  pull_request:
    branches: [ "main" ]
permissions:
  contents: read
  pull-requests: write
jobs:
  breaking-changes:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - run: git fetch --depth=1 origin ${{ github.base_ref }}
      - uses: oasdiff/oasdiff-action/breaking@v0
        with:
          base: 'origin/${{ github.base_ref }}:openapi.yaml'
          revision: 'HEAD:openapi.yaml'
          fail-on: WARN
          github-token: ${{ github.token }}

What each part does:

• on: pull_request runs on every PR into main. To skip PRs that don't touch the spec, add paths: [ "openapi.yaml" ] under pull_request.
• permissions: contents: read to check out the code; pull-requests: write lets the action post the review link as a PR comment. Without it, the link goes to the job summary instead.
• git fetch --depth=1 fetches only the base branch so the action can read the old spec from git. Full history isn't needed.
• base and revision use oasdiff's <git ref>:<path> syntax: the spec on the base branch and the spec in the commit being tested. Each also accepts a plain file path or an http(s) URL.
• fail-on decides whether the job fails (step 2). Leave it out and the action only reports.
• github-token is used to post the review link. On PRs from forks the token is read-only, so the link falls back to the job summary.

Step 2: decide what fails the job

oasdiff sorts each breaking change into a level:

• error: a client that followed the old contract can stop working
• warning: it might; a human should look
• info: shown by changelog, not by breaking

fail-on picks the lowest level that fails the job:

One failure fail-on doesn't control: if oasdiff can't run at all (a spec that doesn't parse, a file missing on one side, an external $ref it isn't allowed to load) the step fails regardless. If the check must never fail the job, also add continue-on-error: true to the step.

Step 3: open a PR and read the output

Here's a sample change: a PR that makes a new query parameter region required on GET /orders, removes the old limit parameter, and drops the required status field from the GET /orders/{id} response.

Annotations on the PR

The action emits GitHub annotations, titled with the oasdiff rule ID. They show up on the Files changed tab and in the run summary:

::error title=new-required-request-parameter,file=openapi.yaml,line=24,col=12::in API GET /orders added the new required `query` request parameter `region`
::error title=response-required-property-removed,file=openapi.yaml::in API GET /orders/{id} removed the required property `status` from the response with the `200` status
::warning title=request-parameter-removed,file=openapi.yaml::in API GET /orders deleted the `query` request parameter `limit`

One gotcha: an added parameter has a line in the new file, so its annotation sits on that line. A removed field has no line in the new file, so GitHub shows it at the top of the file. Read the run summary, not just the diff view.

The full report

The step output breaking only holds the first line (the count). For the whole report, set output-to-file: oasdiff-breaking.txt and upload it as an artifact, or run the CLI locally:

3 changes: 2 error, 1 warning, 0 info
error	[new-required-request-parameter] at openapi.yaml
	in API GET /orders
		added the new required `query` request parameter `region`

error	[response-required-property-removed] at openapi.yaml
	in API GET /orders/{id}
		removed the required property `status` from the response with the `200` status

warning	[request-parameter-removed] at openapi.yaml
	in API GET /orders
		deleted the `query` request parameter `limit`

The side-by-side review link

When there are breaking changes, the action uploads both specs to oasdiff.com (encrypted in CI first; the key stays in the link) and posts a side-by-side review link that expires after 7 days.

If your specs must not leave your runners, set review: false. Nothing is uploaded, and detection and annotations work the same.

Step 4: tune it

Keep settings in a .oasdiff.yaml file in the repo root instead of in every workflow. Inputs in the workflow's with: win over the file.

# .oasdiff.yaml (repository root)
fail-on: ERR
err-ignore: ./oasdiff-err-ignore.txt

When a breaking change is intended and agreed, an ignore file stops it from failing every later run. Each line is the method and path, then the change text as oasdiff printed it:

# oasdiff-err-ignore.txt: one accepted breaking change per line, with the PR that accepted it
# PR #1290: region is required from v2; the two known callers already send it
GET /orders added the new required `query` request parameter `region`

Two more inputs worth knowing:

• allow-external-refs is false in the action (the CLI default is true), so an untrusted PR can't make the runner fetch URLs. Set it to true if your spec is split across files or references external URLs.
• composed: true compares many spec files at once. The side-by-side review isn't available in that mode.

Step 5: what to do with the output

A red job tells the author something broke. It doesn't tell anyone what to do next. For each finding:

• Error: look for an additive change first. Many errors have a non-breaking version: a new optional parameter instead of a required one, a new field next to the old one, a new endpoint or version. Ask for that first.
• Error you keep: name who calls the endpoint. If it has to break, list the teams that call the endpoint and tell them before the merge, with what changes and when. Then add the line to the ignore file in the same PR, so the reason lives in the history.
• Warning: decide and write it down. request-parameter-removed is breaking only if the server rejects input it doesn't know. Check, and put the answer in the PR description.
• No breaking changes: still read the changelog. New endpoints and fields don't break anyone, but consumers may want them. The changelog action lists them.

The step the action can't do

The action answers "what changed?". The hardest line in step 5, "list the teams that call the endpoint", isn't in the spec. If every caller lives in the same repo, the PR reviewers are the right people and you're done. If other teams call your API from other repos, someone has to know who they are.

That's the gap we built Graphwake for: consumer teams subscribe to the endpoints they call, and when a PR changes one in a breaking way, only those teams get an alert in Slack, Microsoft Teams or email. It runs as its own CI step, next to the oasdiff Action, which keeps its own fail-on.

Originally published on graphwake.com. Graphwake is not affiliated with the oasdiff project; oasdiff and its name belong to its authors.

Do you gate merges on breaking changes, or only report them? Curious where teams draw that line.

Inside a company the problem is different. You usually know what you are about to change. What you often don't know is who calls it. The service has been around for three years, two teams have been reorganized since, and somebody's batch job reads a field you are about to rename.

This post is a short playbook for that situation: find the consumers first, tell only them, give a date and an action, then follow up.

Why the usual channels fail inside a company

• Email to everyone gets filtered out within a week. People learn that most of those mails are not about them.
• A changelog only helps the readers who go and look at it. Nobody checks the changelog of an internal service before every deploy.
• A shared channel (#api-changes, #engineering, #general) reaches lots of people who don't care and still misses the one team who does.
• Deprecation headers only help clients that log and alert on them. Most internal clients don't.

All four share the same flaw: they broadcast. Broadcasting works when the audience is small or very motivated. Between 10 or 20 teams it turns into noise, and noise gets ignored.

Step 1: find the consumers

Before you write a single message, build the list of teams that actually call the endpoint. There are three common sources, and each one misses something:

Ask. Post in a channel or ask around. Fast, but you only hear from the people who happen to read it.

Search the code you can see. If most services live in one org, search for the path or the client method:

# GitHub code search (or your own grep across checked-out repos)
"/customers/" "legacyId" org:your-org

This misses services in other orgs, generated clients, and anything that builds URLs dynamically.

Check gateway or access logs. If traffic goes through a gateway, group recent calls by caller:

-- example: access logs in a warehouse table
SELECT client_id, count(*) AS calls
FROM gateway_access_log
WHERE path_template = '/customers/{id}'
  AND ts > now() - interval '30 days'
GROUP BY client_id
ORDER BY calls DESC;

This shows who called recently, but not who calls once a quarter, and a client_id is not a team.

Combine all three and you will get most of the way. The most reliable source, though, is a list the consumers keep themselves: each team subscribes to the endpoints it calls. The producer doesn't have to guess, and the list stays current when teams change.

Step 2: tell those teams only

Message the consumer teams directly, in the channel they already read: their team channel, their on-call alias, a ticket in their board. Not a company-wide channel.

A short message sent to five people gets read. A long one sent to five hundred does not.

Step 3: give a date and an action

Every notice needs two things:

• A date. A change without a date gets ignored, because there is no reason to act now.
• An action. A change without a concrete "do this instead" gets postponed, because figuring it out is extra work.

How far ahead? Long enough for the slowest affected team to change and ship. Agree on the date with the teams you found in step 1 rather than announcing it.

A notice template

Copy this, fill it in, send it to the teams from step 1:

What changes:   GET /customers/{id} no longer returns legacyId
                (removed from the response).
Who is affected: teams that read legacyId: [list the teams].
When:           merged on [date]; deployed on [date].
What to do:     use id instead. Before/after example: [link].
Questions:      reply in this thread or ask [owner].

Five lines. Each one answers a question the reader would otherwise have to ask you.

Step 4: confirm and follow up

Keep a simple list: who was told, who answered, who has moved. When the date arrives, check that every affected team has either shipped the change or asked for more time. Only then deploy.

That list is also useful later. When something does break, the first question in the incident review is "did they know?", and you will have the answer.

Doing this automatically

Steps 1 and 2 are the tedious part, and they repeat for every breaking change. We built Graphwake to do them from your OpenAPI spec:

• Consumer teams subscribe to the endpoints they call, so the consumer list stays current.
• When a pull request introduces a breaking change, the alert goes only to those teams, in Slack, Microsoft Teams or email.
• The alert names the service, the endpoint, its owner and what changed.

It works from a published spec URL or one CI step. Steps 3 and 4, the date and the conversation, are still yours.

Originally published on graphwake.com. How does your team handle this today? A channel, a changelog, or just asking around? We'd like to hear what actually works.