Experimently Documentation

Run the stack and create your first experiment.

• Quick Start—Run the stack and create your first experiment
• Core Concepts—Feature flags, experiments, variants, and metrics
• Architecture Overview—How the components fit together, and how data flows between them
• FAQ—Common questions answered
• Modules & Profiles—What the core profile includes, what each module adds, and how to run the full profile

Create flags, target them with rules, and roll them out in stages.

• Creating Feature Flags—Set up your first flag
• Targeting & Rules—The operators a targeting rule can use, with examples
• Gradual Rollouts—Staged rollout schedules
• Safety Monitoring—Roll a flag back when its error rate or latency breaches a threshold

Create and start experiments, read their results, and the statistics behind them.

• Running Experiments—End-to-end experiment lifecycle
• Dashboard and API—Which capabilities have a dashboard screen, and which work through the API only for now
• Statistical Methods—Sequential testing with mSPRT
• CUPED—Variance reduction from each user's own pre-experiment events (API only)
• Multi-Armed Bandits—Thompson Sampling, UCB1, Epsilon-Greedy (create a bandit through the API)
• Split URL Testing—Full profile: store split URL experiments and preview the URL a user id hashes to. The edge router does not split traffic yet.
• Mutual Exclusion Groups—Keep users out of conflicting experiments, and a share of new users out of every experiment
• Live Results Streaming—Live experiment results over WebSocket: a snapshot on connect, then a fresh one every 30 seconds (by default), whatever the client sends.

Compare model variants on a task; the winner has the best business metric when every variant has one, and otherwise the best LLM-judge score.

• Overview—Supported providers, task types, variant assignment, judge scoring, cost estimation and the analysis
• Quick Start—Compare GPT-4o with Claude 3.5 Sonnet (steps 3–5 need provider API keys)

Client libraries for web, mobile and server. The JavaScript SDK is on npm and the Python SDK is on PyPI.

• JavaScript SDK—Browser and Node.js
• Python SDK—A synchronous Python client with no dependencies
• Java SDK—Spring Boot 3 auto-configuration starter; thread-safe client with a TTL cache
• React SDK—A context provider, hooks and a higher-order component
• Go SDK—A Go client for services: standard library only, goroutine-safe, context-aware.
• iOS Swift SDK—Native Swift SDK for iOS 14+ and macOS 11+, with an async/await API.
• Android Kotlin SDK—Native Kotlin SDK for Android (minSdk 21). Coroutines, OkHttp, Compose examples.
• Flutter SDK—Dart SDK for Flutter (iOS, Android, web and desktop). Results are decided by the server, with an offline fallback.
• React Native SDK—useFlag and useExperiment hooks, Provider, AsyncStorage offline support.
• MCP tool manifest—Describes the API's tools for AI assistants.
• OpenFeature Provider—OpenFeature (a CNCF project) providers
• Edge SDK—For Cloudflare Workers, Vercel Edge and Deno Deploy. Zero runtime dependencies; results cached in memory or KV.
• Ruby SDK—Ruby gem with zero runtime dependencies, a thread-safe TTL cache and Net::HTTP.
• PHP SDK—For PHP 8.1+, installed with Composer; it needs only ext-curl and ext-json.
• .NET SDK—Targets netstandard2.1 and net6.0; System.Text.Json and HttpClient, no other dependencies.
• Elixir SDK—For Elixir and Phoenix: OTP's :httpc and Jason, with a GenServer-backed ETS cache.

How to call the REST API: authentication, and the endpoints with example requests.

• Authentication—API keys for SDKs, bearer tokens for people
• Experiments—List experiments, get one, and get its results
• Feature Flags—List flags, a user's flags, and archived flags
• Compliance & Audit—Full profile: signed audit events and evidence-report export
• Integrations—Jira, Salesforce, GitHub webhooks
• All Endpoints—Endpoint reference, with example requests and error responses

The AWS services Experimently runs on, the webhooks it receives, and the alerts it sends.

• AWS—How the platform uses ECS Fargate, Aurora, ElastiCache and DynamoDB
• Jira—Inbound webhooks, signed or with a shared secret
• Salesforce—Inbound webhooks from a Flow or Apex callout
• GitHub—Inbound webhooks, each checked by its HMAC-SHA256 signature
• Slack & Email—Slack messages for scheduled experiment changes and rollout stage advances; email only from the test route

Audit records, compliance reports, roles, API keys and single sign-on.

• Compliance audit logging—An append-only record of changes to experiments and flags; signing and export in the full profile
• Compliance evidence—Full profile: SOC 2 and ISO 27001 reports and an export of the audit events
• Audit Logging—Who changed what through the API; no API edits or deletes an entry
• RBAC—Admin, Developer, Analyst and Viewer decide what a user can do; custom roles (full profile) are stored but not applied yet
• API Key Management—Create, expire and delete keys; the sdk:ruleset scope for local evaluation
• SSO / SAML / OIDC—Full profile: sign in through Okta, Google or GitHub (OIDC). SAML 2.0 sign-in is not available yet; use OIDC.
• HIPAA controls—Full profile: PHI encryption (Fernet), a 6-year retention marker on PHI audit records, BAA records and the data-residency configuration.

Run Experimently yourself: Docker Compose on one machine, Helm on Kubernetes, or a CDK app that deploys to AWS.

• AWS CDK Deployment—CDK app for ECS Fargate, Aurora and ElastiCache
• Docker Compose—The published images on one Docker host: deploy/compose/compose.yml, and a README that says how to run it and lists its settings
• Kubernetes (Helm)—The Helm chart in charts/experimently: a one-time secret step, then one install command
• Environment Variables—Configuration reference
• Database Migrations—Alembic migration guide
• Monitoring—CloudWatch dashboards, Prometheus metrics

Walkthroughs and method guides.

• Your First A/B Test—End-to-end experiment walkthrough
• Bayesian Experimentation—Beta-Binomial posteriors and stopping rules
• Guided Experiment Builder—Guided setup in the dashboard, and the wizard API
• Interaction Detection—Beta: find experiments that share users, and whether their effects interact

How power analysis, post-stratification and FDR correction work.

• Power Analysis & Sample Size—Pre-experiment planning: MDE, power and runtime estimation
• Post-Stratification—The method, explained; not available through the API yet
• FDR Correction—Benjamini-Hochberg false discovery rate correction for multiple metrics

Group members into teams, projects or product areas with workspace roles. Workspaces do not limit access to experiments or flags; that is by platform role.

• Workspace Overview—What workspaces do and do not do, workspace roles and invitations
• Quickstart—Create a workspace, add your team to it, and connect an SDK