# Admin UI Bundled as embedded resources in the `CommerceMind.Nexus.UI` NuGet package is an admin UI. It uses the [API](../api/index.md) for all data and requests, so anything that's possible to do in the admin UI is possible to do with the API. Register the Admin UI and tell it where the Nexus API is hosted: ```cs using CommerceMind.Nexus.UI.Extensions; builder.Services .AddNexus() .AddApi() .AddAdminUI(options => { options.ApiBaseUrl = "/"; }) .Build(); var app = builder.Build(); app.UseNexusAdminUI(); ``` When the API and Admin UI are hosted in the same application, set `ApiBaseUrl` to the same value as `NexusApiOptions.BasePath`. Feature discovery is served by the API, including when the UI is hosted separately. Menu customization is supplied by the UI host when it serves the page. If you deploy Nexus to multiple environments such as dev/test/stage/prod/etc you can add an environment name that is shown in the UI, as well as add an info notice bar with any message you want. Both of these settings are used in the demo environment described below to see it in action. ```cs builder.Services.AddNexus().AddAdminUI(options => { options.ApiBaseUrl = "/"; options.EnvironmentName = "Dev"; options.InfoNotice = "This is a note to anyone using the UI"; }); ``` The UI uses the API host's `/api/enabled-features` response to show available features, together with the menu supplied by the UI host. If you want to control the menu you can implement the menu loader to return which menu items you want: ```cs builder.Services.AddNexus().AddAdminUI(options => { options.ApiBaseUrl = "/"; options.MenuItemsLoader = httpContext => Task.FromResult(new List { NexusAdminUIMenuItems.HealthChecks, NexusAdminUIMenuItems.Jobs, NexusAdminUIMenuItems.Queues, NexusAdminUIMenuItems.Functions, NexusAdminUIMenuItems.Statistics, }); }); ``` The menu loader is an async lambda that gets an HTTP request passed in that lets you have different menu items for different users. The order in the returned list is respected. ## Admin UI versions The new version of the admin UI is enabled by default. You can temporarily switch back to the legacy admin UI by setting `UseLegacyVersion` to true: ```cs builder.Services.AddNexus().AddAdminUI(options => { options.ApiBaseUrl = "/"; options.UseLegacyVersion = true; }); ``` `UseLegacyVersion` and the legacy admin UI are retained as a temporary fallback. They will be removed in a future major version. ## Jobs, activation, and processing controls The jobs list and job details show live progress reported by `IJobProgress`. Queue jobs do not generate progress automatically unless [automatic progress](../queues/jobs.md#automatic-progress-reporting) is enabled. A readiness-driven job can be waiting for messages rather than waiting for a cron run; clearing a queue job's saved schedule switches to that mode. A cron schedule makes it scheduled-only. Both UI versions show the optional queue-job schedule and activation delay. The new UI also shows distributed processors on their queue pages, including per-instance worker activity, in-flight messages, and pause/resume/wake controls. They do not create job runs in the jobs list. The Processing tab saves [per-queue processing overrides](../queues/processing-settings.md). Clearing an override restores the code default; **Use code defaults** resets all overrides. Jobs apply changes on the next run, while distributed pools finish current work before rebuilding. The legacy UI does not expose these new processing controls. ## Hosting the UI on its own If you want to separate the Admin UI from the API you can host the Admin UI on it's own. Just install the NuGet package `CommerceMind.Nexus.UI` in a separate ASP.NET Web app and in your `Program.cs` you add: ```cs builder.Services .AddNexus() .AddAdminUI(options => { options.Path = "nexus-ui"; // Optional, default is "nexus" options.ApiBaseUrl = "https://your-nexus-api-url.com"; }) .Build(); var app = builder.Build(); app.UseNexusAdminUI(); ``` This will serve the Admin UI without needing a database connection or any other access to Nexus but the API. ## Demo environment There's a demo environment that's deployed to a free tier Azure Web App here:\ [https://commerce-mind-nexus-a9bsbwgth3cgfnde.swedencentral-01.azurewebsites.net/admin/](https://commerce-mind-nexus-a9bsbwgth3cgfnde.swedencentral-01.azurewebsites.net/admin/) You can click on any buttons you like, start any job or do anything with the queues to get a feel for what the system does. :::note[Demo environment] Since the demo environment is using a free tier Azure Web App the scheduler is paused when nobody is using the API or admin UI which means that the jobs won't run exactly according to schedule. ::: ## Authentication See [API docs about authentication](../api/index.md#authentication). ## Authorization See [API docs about authorization](../api/index.md#authorization). --- # Nexus HTTP API A REST API for all Nexus features are included, and it's what the admin UI uses. The API is not enabled by default but all you need to do to enable it is: ```cs builder.Services.AddNexus().AddApi(options => { // The default options are: options.BasePath = "/"; options.RegisterHealthChecksAsAspNetCoreHealthChecks = true; options.AllowAnonymousApiAccess = true; }); ``` When `BasePath` is `/` it means that the API is accessible on `https://servername/api` and if you set it to `/mybasepath/` the API is accessible on `https://servername/mybasepath/api`. `BasePath` no longer configures the separately packaged Admin UI. When hosting them together, pass the same value to `NexusAdminUIOptions.ApiBaseUrl`; see the [Admin UI documentation](../admin-ui/index.md). ## Authentication One of the options above is if the system should allow anonymous API access or not. If you configure the API not to allow it the system does not do any authorization though and leaves it up to you to authorize the user through eg a cookie (see more about authorization below). The system will check if `HttpContext.User.Identity.IsAuthenticated` and otherwise respond with HTTP status `401` on any API requests. If you wish to add login to the Admin UI you should setup a middleware on the admin route which checks if the user is signed in or not and redirect to/display a login page. Something like this: ```cs // Program.cs builder.Services.AddNexus() .AddApi(options => options.AllowAnonymousApiAccess = false) .AddAdminUI(options => options.ApiBaseUrl = "/"); var app = builder.Build(); app.UseAuthentication(); app.UseAuthorization(); app.Use(async (context, next) => { if (context.Request.Path.StartsWithSegments("/nexus") && context.User.Identity?.IsAuthenticated != true) { context.Response.Redirect("/login"); await context.Response.CompleteAsync(); } else { await next(); } }); app.UseNexusAdminUI(); ``` If you then implement the `/login` route to show a login form that redirects back to `/nexus` when the user has logged in your Nexus admin UI and API is fully protected. ## Authorization Nexus offers a very granular level of authorization on endpoints that perform update/create/delete actions. When initializing the Nexus API you can pass an async lamda to authorize an HTTP request like this: ```cs // Program.cs builder.Services.AddNexus().AddApi(options => { options.IsReadOnly = async (httpContext, nexusEndpoint) => { if (nexusEndpoint == NexusEndpoint.DeleteAllQueueItems) { var user = await httpContext.RequestServices.GetRequiredService().GetCurrentUserAsync(); // Only allow admins to nuke all queue items return !user.IsAdmin; } return false; }; }); ``` The `NexusEndpoint` class contains all endpoints that the Nexus API contains which mutates anything. If you want to build a dynamic registry of which users are allowed to use which endpoints you can use the `NexusEndpoint.AllNexusEndpoints` list to base it on. The list is generated and the static name of an endpoint is the same as the static name. That is, this will always be true: ```cs nameof(NexusEndpoint.Xyz) == NexusEndpoint.Xyz.Name ``` The implementation here is of course made up and your implementation would look differently. Any update/create/delete request sent to the API where the `IsReadOnly` lambda returns true will fail with a `403` status. Note that the admin UI will still display all editing controls but will fail with an error message if a user tries to do something unauthorized. ## List responses and identities `GET /api/jobs` returns `{ "jobs": [...], "categories": [...] }`. There is no separate `GET /api/jobs/categories` endpoint. A queue job's schedule can be null; use its activation/waiting state rather than assuming it always has a next cron run. `GET /api/queues` omits counts by default. Pass `queueNamesWithCounts` for selected queues or `includeCountsForAll=true` to request all counts. Missing counts mean not loaded, not zero. Historical job runs are identified by `correlationId`; extra-run IDs are 64-bit integers. Custom clients must also recognize `AwaitingRelease` and `statusAfterRelease`. The start endpoint follows `IJobStartRequester`: an immediate request with no parameters can be best-effort. Specify an explicit UTC `startAt` to persist an extra run, or use `IJobMetaDataRepository.ScheduleExtraRunAsync()` in application code. See [job starts](../jobs/index.md#starting-a-job-programatically). Queue creation payloads no longer have `startProcessing`. Ready messages follow the queue's configured [processing mode](../queues/activation.md). [Processing settings](../queues/processing-settings.md) and distributed-processor status/pause controls are available on queue-specific endpoints. ## Feature discovery `GET /api/enabled-features` reports the API host's configured features and is served by the API package. A separately hosted Admin UI uses this endpoint on its configured API base URL. Menu customization belongs to the UI host's `MenuItemsLoader`. ## Endpoints Besides the `NexusEndpoint` class for mutative endpoints you can see a full list of the API endpoints for the Nexus API in the [API reference](/api/reference/nexus-api/). The [OpenAPI JSON document](/api/swagger.json) is also available for client generation. Nexus also has an optional MCP HTTP server for agents and tools. See [Nexus MCP](./mcp.md) for setup and available tool groups. There's a demo environment that's deployed to a free tier Azure Web App here:\ [https://commerce-mind-nexus-a9bsbwgth3cgfnde.swedencentral-01.azurewebsites.net/swagger/index.html](https://commerce-mind-nexus-a9bsbwgth3cgfnde.swedencentral-01.azurewebsites.net/swagger/index.html) You can send requests to any endpoint you'd like to get a feel for what the system does. Since it's a free tier Azure Web App the scheduler is paused when nobody is using the API or admin UI which means that the jobs won't run exactly according to schedule. --- # Nexus MCP `CommerceMind.Nexus.Mcp` exposes Nexus operations as an MCP HTTP server. It is intended for tools and agents that need to inspect or operate Nexus jobs, queues, functions, health checks, and instances without calling the REST API directly. The MCP package is opt-in. Installing or enabling `CommerceMind.Nexus.Api` does not automatically add the MCP server. ## Setup Install/reference the MCP package in the application that hosts Nexus: ```xml ``` Then register the MCP services and map the endpoint: ```cs using CommerceMind.Nexus; using CommerceMind.Nexus.Api.Extensions; builder.Services .AddNexus() .AddApi() .AddMcp(options => { // This is the default value options.McpEndpoint = "nexus-mcp"; // Only allow agents to read. Set this to false to allow agents to update, delete and run jobs and functions. // The default is true. options.ReadOnly = true; }) .Build(); var app = builder.Build(); app.MapControllers(); app.MapNexusMcp(); ``` ## Security The MCP endpoint is an ASP.NET Core endpoint, not an MVC controller. The Nexus API authorization filter does not automatically protect it. Protect it using normal ASP.NET Core middleware and endpoint conventions: ```cs app.UseAuthentication(); app.UseAuthorization(); app.MapNexusMcp().RequireAuthorization(); ``` Many MCP tools can mutate Nexus state, such as starting jobs or updating/deleting queue items. The typical setup is to require an API key for the MCP endpoint. --- # Nexus vs Function as a Service If you're reading the this documentation you might be thinking "well I can do this with scheduled Azure Functions and Azure Service Bus without any need for something like Nexus". And you're right, you can do that and that setup is not bad at all. But Nexus has features that the Azure Function + Service Bus setup doesn't offer out of the box. Instead of seeing Nexus as a replacement for it you can see it as a complement. If you're already using Azure Functions you can have a function that executes Nexus jobs and functions since Nexus doesn't require you to run it as a [continuous background service](./general/hosting.md). Something like this: ```cs public class NexusInvocation(IServiceScopeFactory scopeFactory) { public async Task RunAsync(CancellationToken cancellationToken) { await using var scope = scopeFactory.CreateAsyncScope(); var jobs = scope.ServiceProvider.GetRequiredService(); var functions = scope.ServiceProvider.GetRequiredService(); var jobResults = await jobs.RunAllJobsPendingSinceLastCallAsync(cancellationToken); var functionResults = await functions.RunAllFunctionsPendingSinceLastCallAsync(); return $"{jobResults.Count} Nexus jobs and {functionResults.Count} Nexus functions executed"; } } ``` Call this operation from the hosting platform's trigger and configure both `JobsOptions.UseBackgroundService` and `NexusFunctionsOptions.UseBackgroundService` to false. See [hosting](./general/hosting.md) for registration. The operation creates a scope because both executors are scoped. For periodic hosts, persist required job starts through `IJobMetaDataRepository.ScheduleExtraRunAsync()` rather than relying on immediate parameterless `IJobStartRequester` events; those events are handled only by continuous job loops. Readiness-driven queue jobs are reconciled when the job executor is invoked. Do not register distributed queue processors in a host that stops between invocations. You can invoke this operation as often as you like and Nexus will make sure to call the jobs and functions that should run. If you schedule the Azure function to run every five minutes and you have Nexus jobs that are scheduled every minute they will only run every five minutes. You should schedule the Azure function to run as often as you want your most often running Nexus job to run. :::note[Cost efficiency] Running Nexus as a scheduled Azure function can be a cost efficient way of running functions and background jobs in test environments since it doesn't require a server that's always running. ::: ## Benefits with Nexus jobs When everything is running smoothly Azure functions and Nexus are quite equal. But Nexus starts to shine when things aren't. When you get a call from a business stakeholder and she tells you to stop the order export you can quickly go in to the admin UI and [disable the job](./jobs/index.md#disabling-a-job) or even guide the stakeholder how to do it herself. Empowering non-developers with regards to background jobs and queues is one of Nexus strengths. With [health status](./general/monitoring.md) and [job logs](./jobs/logs.md) visible in the admin UI it's possible for almost anyone to get at least a basic understanding of why things are failing and help out in troubleshooting. Another big benefit of Nexus is how easy it makes local development. You're not dependant on accessing cloud functionality to build jobs and with SQLite you don't even need a database server. Minimizing the differences between how code runs locally vs in production is very valuable. ## Benefits with Nexus functions Many of the benefits with Nexus jobs applies to Nexus functions as well but they deserve a special mention since the name can make them seem to do the same thing as Azure Functions. The big difference is how easy it is to pass arguments in a type safe way to a Nexus function compared to an Azure function. ## Nexus queues vs Azure Service Bus queues Azure Service Bus queues are great and have a lot of functionality. One of the strength of Azure Service Bus queues is that it can handle a very large number of messages. For some workloads raw broker throughput is the main requirement; for others inspection of individual messages and controlled retries matter more. Instead you want more insight into each and every message for these processes. A trade-off in being able to handle that kind of volume is that Azure doesn't offer the same kind of introspection into queues as Nexus does. You can read, filter, paginate and update Nexus queues either through the UI, the API or in your favorite SQL management application. You can use the built in JSON functionality of your database to find messages by properties in the JSON message. Since Nexus makes a different trade-off than Azure it lets Nexus queues offer features that aren't possible with Azure queues. Such as [virtual queues](./queues/virtual.md), [idempotent messages](./queues/index.md#idempotent-messages) and [to store processed messages](./queues/retention.md). Storing processed messages lets you build sophisticated change tracking since you get access to the previous version of the message to let you diff them. If you want to you can also use the `CommerceMind.Nexus.Azure` NuGet package to read [from an Azure queue and add them to a Nexus queue](./queues/azure.md). For high-throughput workloads, Nexus can also process directly from [external pending storage](./queues/external-pending.md), including Azure Service Bus topics. [Distributed processors](./queues/processors.md) let every continuously running instance contribute workers. These options keep Nexus's queue controls while changing where pending messages live and how processing scales. --- # Database The system uses a SQL database to store meta data and queue messages. At the time of writing we support SQLite, Postgres and SQL Server. Feel free to reach out to us if you need support for another database engine like MySQL or MariaDB. ## Configuring which database to use The support for the different databases exists in their own NuGet packages: ``` CommerceMind.Nexus.Postgres CommerceMind.Nexus.SqlServer CommerceMind.Nexus.Sqlite ``` Install the package(s) you need before you initialize it below. After you've installed the db package(s) you open your `Program.cs` and go to these lines: ```cs builder.Services .AddNexus() .AddSingleServerInstanceEvents() // Use a shared broker when running multiple instances .AddFunctions() .AddJobs() .AddQueues() // Either: .AddPostgresConnection() // Or: .AddSqlServerConnection() // Or: .AddSqliteConnection() // This should be the last thing you do, after you've finished configuring Nexus .Build(); ``` By default the system will call [`IConfiguration.GetConnectionString("postgres")`](https://docs.microsoft.com/en-us/dotnet/api/microsoft.extensions.configuration.configurationextensions.getconnectionstring?view=dotnet-plat-ext-6.0), [`IConfiguration.GetConnectionString("sqlserver")`](https://docs.microsoft.com/en-us/dotnet/api/microsoft.extensions.configuration.configurationextensions.getconnectionstring?view=dotnet-plat-ext-6.0) or [`IConfiguration.GetConnectionString("sqlite")`](https://docs.microsoft.com/en-us/dotnet/api/microsoft.extensions.configuration.configurationextensions.getconnectionstring?view=dotnet-plat-ext-6.0) to get the connection string to the database. If that doesn't work for you can register a lamda to provide your connection string like this: ```cs builder.Services.AddNexus().AddPostgresConnection(options => { options.ConnectionStringLoader = serviceProvider => { // Find and return your connection string here }; }); // Or: builder.Services.AddNexus().AddSqlServerConnection(options => { options.ConnectionStringLoader = serviceProvider => { // Find and return your connection string here }; }); // Or: builder.Services.AddNexus().AddSqliteConnection(options => { options.ConnectionStringLoader = serviceProvider => { // Find and return your connection string here }; }); ``` :::note[Using a ConnectionStringLoader] When you use `ConnectionStringLoader` you need to make sure that it's a very fast operation as the loader will be called every time Nexus runs a database query. You need to make sure to cache the value if you fetch it from an external store such as Azure KeyVault. ::: ## Controlling the connection By default Nexus will open and close database connections for almost every query, and rely on the underlying connection pooling. In some cases you might want to have more control over the connection such as if you want to include the Nexus queries inside a transaction. One option is to use a `DbConnectionScope` like this: ```cs public async Task RunAsync( IEnqueuer enqueuer, IQueueReadinessNotifier readinessNotifier) { // CreateDbConnection returns an application-owned DbConnection. await using var connection = CreateDbConnection(); await connection.OpenAsync(); await using var transaction = await connection.BeginTransactionAsync(); using (new DbConnectionScope(connection, transaction)) { await enqueuer.EnqueueAsync(new SomeMessage()); await transaction.CommitAsync(); } // The caller owns this explicit commit, so notify after it succeeds. await readinessNotifier.NotifyAsync(enqueuer.QueueName); } ``` `DbConnectionScope` is in `CommerceMind.Nexus.Abstractions.Core.Database`. Compatible Nexus database operations inside the block reuse the supplied connection/transaction. It does not make remote APIs, external brokers, or in-memory updates transactional. The caller owns commit/rollback and should notify readiness after a successful explicit commit; the scope does not provide a general after-commit hook. If the [EF integration](../queues/entity-framework.md) is enabled, use its scoped unit of work instead of this standalone-enqueue example: queue operations are deferred until EF saves/commits. EF already requests readiness after commit. Use `ImmediateQueueOperationScope` only for deliberately independent operations, not to bypass entity-plus-message atomicity. ## Database initialization Nexus initializes its tables and queue schemas before background processing begins. SQL Server and PostgreSQL serialize initialization with session-owned database locks. SQLite is intended for simple/local use and does not coordinate concurrent initializers; initialize it from one process. V3 stores completed migration manifests and per-queue schema definitions in reserved `Nexus.Startup.*` key-value records. Unchanged starts skip migration-history and queue table/index/column checks. A new queue or changed filterable properties still trigger their own initialization, even if regular migrations have already run. Changed filters are backfilled from persisted messages, so those messages must remain readable by the updated contract. Interrupted backfills retain a plan and can resume on the next start. Keep Nexus-owned schema and initialization records intact. Nexus no longer inspects or automatically repairs an unchanged table after manual schema edits. Coordinate instances when deploying incompatible filter/schema changes. Application migrations belong in regular migrations; custom idempotent migrations still run at every startup. When upgrading from v2, stop all v2 instances first. V3 renames the persisted job tables and rebuilds queue lease indexes. Allow initialization time and extra database capacity for large queues; see the [upgrade guide](upgrade-guide.md#database-initialization-in-v3). A package downgrade alone does not reverse these changes. ## Transient errors Nexus has built-in retries of transient errors using [Polly](https://github.com/App-vNext/Polly) but if you want to you can add custom retry logic by creating your own implementation of the `ITransientDatabaseErrorRetryer` interface. ## Database maintenance Nexus includes two jobs that are enabled by default. One job is called `RebuildDatabaseIndexes` and will rebuild database indexes for Postgres and SQL Server. It's scheduled to run daily at midnight in `JobsOptions.ScheduleTimeZone` (UTC by default). If you're using SQLite this job won't do anything as it's not needed to rebuild SQLite indexes. By default this job will only rebuild indexes in tables created by Nexus. If you want it to rebuild indexes for all tables in the database you can can pass an option to `AddNexus().AddPostgresConnection()` or `AddNexus().AddSqlServerConnection()`: ```cs builder.Services.AddNexus().AddPostgresConnection(options => { options.LimitDatabaseMaintenanceToNexusTables = false; }); // Or: builder.Services.AddNexus().AddSqlServerConnection(options => { options.LimitDatabaseMaintenanceToNexusTables = false; }); ``` The other job is called `RunDatabaseMaintenanceJob` and is also enabled by default and but not scheduled. For Postgres that will run `VACUUM` and then `ANALYZE`. If you're using SQL Server this job won't do anything as SQL Server has automatic update of query statistics. If you're using SQLite this job will not do anything either. ## Remove database maintenance jobs If you want to completely hide these jobs you can pass a filter lambda when you register the jobs system to exclude these jobs like this: ```cs builder.Services.AddNexus().AddJobs(options => { options.JobFilter = jobType => jobType.Name != "RebuildDatabaseIndexesJob" && jobType.Name != "RunDatabaseMaintenanceJob"; }); ``` If you want to run the jobs but do something different than the defaults you can create your own implementation of the `IDatabaseMaintenanceService` interface and register it with the service provider like this: ```cs builder.Services.AddScoped(); ``` ## Using another database engine If you want to use another database engine such as MySQL or MariaDB you can always implement it and open a PR to the Github repository. If you don't have access yet to the repository just ping us! Another option is of course to hire us to implement it for you. --- # Dependency injection and scopes Most operational Nexus services are scoped in v3. Jobs, functions, queue operations, health checks, and integrations can use scoped application dependencies, including a `DbContext`. The scope that resolves a service owns its lifetime. ## Calling Nexus in a request or application operation Controllers and other request-scoped application services can inject Nexus dependencies directly: ```cs public class OrderPublisher(IEnqueuer enqueuer) { public Task PublishAsync(OrderMessage message) => enqueuer.EnqueueAsync(message); } builder.Services.AddScoped(); ``` The same applies to transient services resolved within that scope. A transient service resolved by a singleton does not make a scoped dependency safe to capture. Choose an application's lifetime based on its requirements. Stateless or shared singleton dependencies can remain singleton when they do not capture scoped services. Nexus's process-wide interfaces such as `INexusInstance`, `IInstanceEventBroker`, `IGracefulShutdownService`, and `IDateTime` also remain singleton. ## Calling Nexus from a singleton or hosted service Inject `IServiceScopeFactory` and create a scope around each independent operation. Resolve the Nexus service from the scope, and keep the scope alive until its asynchronous work is complete: ```cs public class BackgroundPublisher(IServiceScopeFactory scopeFactory) { public async Task PublishAsync(OrderMessage message) { await using var scope = scopeFactory.CreateAsyncScope(); var enqueuer = scope.ServiceProvider.GetRequiredService>(); await enqueuer.EnqueueAsync(message); } } ``` This example assumes no EF unit of work is deferring the enqueue. Do not cache the resolved enqueuer, return it to a caller, or dispose the scope while work is still running. The same pattern applies to `INexusFunction`, job repositories, queue readers/updaters, and other scoped Nexus dependencies. Do not resolve them from `app.Services` or another root provider. ## Entity Framework unit of work When using the [EF integration](../queues/entity-framework.md), resolve the context and enqueuer from the same scope. Database queue messages are saved with the entity transaction, and in-memory additions happen after commit. Use `SaveChangesAsync()` and asynchronous commits. A new scope created by a singleton does not join an arbitrary caller's context or transaction. For a standalone enqueue that must persist without saving a context, an infrastructure publisher can deliberately use `ImmediateQueueOperationScope`: ```cs await using var scope = scopeFactory.CreateAsyncScope(); using var immediate = new ImmediateQueueOperationScope(); var enqueuer = scope.ServiceProvider.GetRequiredService>(); await enqueuer.EnqueueAsync(message); ``` Normal entity-plus-enqueue operations should keep the EF transaction boundary. Remote API/external-pending delivery cannot roll back with that transaction; use an outbox database queue when delivery must follow the entity commit. ## Execution scopes and concurrency | Execution | Scope and instance behavior | | --- | --- | | Function call | Target service resolved in a fresh scope for that execution | | Queue job | Job resolved in a job execution scope; processing threads share the job instance | | Distributed processor | Fresh scope and processor instance per leased chunk; calls on that instance are sequential | | Application instance-event handlers | Resolved in a scope for handling the application event | | Nexus scoped hosted-service wrapper | Holds a scope for the worker lifetime, rather than creating a scope per iteration | Scoped does not mean thread safe. Queue jobs using several processing threads must protect shared state and must not concurrently use one EF context. Separate processing executions can also share singleton application dependencies, which must be safe for concurrent use. ## Custom registrations and health publishers Register custom serializers, repositories, middleware, storage, and other operational replacements with lifetimes appropriate to their dependencies. A singleton replacement cannot inject scoped Nexus services simply because the built-in implementation is now scoped. The .NET health publisher host expects singleton `IHealthCheckPublisher` registrations. A publisher that needs scoped Nexus dependencies can use Nexus's adapter: ```cs using CommerceMind.Nexus.Abstractions.Core.DependencyInjection; builder.Services.AddScopedHealthCheckPublisher(); ``` The adapter lazily creates and retains one scope/publisher until disposal, preserving publisher state between reports. It does not create a fresh scope for each report. If an operation needs short-lived resources such as a context per publication, create that inner scope explicitly. A publisher that only uses singleton dependencies can still be registered as a singleton directly. Enable DI scope validation in development to catch root-provider resolution and singleton captures, including captures through intermediate transient services. --- # Nexus v2 to v3 upgrade skill Use this skill when upgrading an application and its libraries from Nexus v2 to v3. It is intended for an LLM working in the consuming application's repository. The [overview](upgrade-guide.md#from-v2-to-v3) explains the main changes; this file supplies migration instructions and the details needed to preserve behavior. The reference is Nexus 3.0.1. Check the actual target package's interfaces when upgrading to a later v3 version. ## Working without the Nexus source repository GitHub access is optional. The migration rules and selected contract excerpts below are included so this skill can be used in a consuming repository without a Nexus checkout. Links to other pages and GitHub are supplementary references, not prerequisites. When an application-specific integration needs more detail than this guide supplies: 1. Identify the exact target package versions from the application's package references and restored `obj/project.assets.json`. Restore the packages through its configured NuGet feeds if needed. Do not assume the latest source on `main` matches those binaries. 2. Inspect the target package's XML documentation, IDE metadata view, or assemblies. `project.assets.json` records the resolved versions, package folders, and assembly paths. `dotnet nuget locals global-packages --list` also locates the global package cache; build output may already contain the required DLLs. 3. **Decompile the exact v3 assemblies when necessary**, using an available .NET decompiler or the IDE's decompilation support. Start with `CommerceMind.Nexus.Abstractions.dll` for interfaces and records, `CommerceMind.Nexus.dll` for core behavior and registrations, and the applicable `CommerceMind.Nexus.EntityFramework.dll`, database-provider, or broker assembly for integration behavior. Follow the relevant implementation and call sites, not just its public signature. See the inspection map at the end of this guide. 4. Record the package version and whether a conclusion came from this guide, package metadata, decompiled implementation, or matching source. If neither source nor binaries are available, identify the exact unverified contract and continue independent work; do not invent a signature or claim the integration was validated. Do not require the user to obtain GitHub access just to inspect APIs available in their installed packages. Decompilation is for understanding the dependency; update the consuming application rather than copying or modifying Nexus internals. ## Outcome and boundaries Produce a working v3 application, an account of intentional behavior changes, and deployment instructions for its actual database and brokers. Preserve existing queue names, job names, message contracts, schedules, retention policies, and transaction boundaries except for changes requested by the user or agreed through the questions below. Distributed processors, named queues, Redis activations, and local-time schedules are optional features. Do not adopt them merely to complete the upgrade. Keep existing queue jobs unless the application needs a different execution model. Code changes and local validation do not imply permission to migrate a live database, drain a broker, or deploy. Prepare those actions for the environment's normal release process. A package downgrade alone is not a rollback after v3 database migrations or broker changes. ## 1. Inventory the application Read the repository's instructions and identify every host, shared registration library, producer, worker, API client, and separately hosted Admin UI that uses Nexus. Inspect project files, central package versions, SDK selection, container images, CI, deployment manifests, and configuration sources. Useful searches, adapted to the repository: ```text rg -n 'CommerceMind\.Nexus|TargetFramework|TargetFrameworks|net8\.0|net9\.0' -g '*.csproj' -g '*.props' -g 'global.json' -g 'Dockerfile*' rg -n 'AddNexus|AddScheduledJobs|ScheduledJob|EnqueueAndStartProcessing|IQueueProcessingRequester' -g '*.cs' rg -n 'AddSingleton|BackgroundService|IHostedService|GetRequiredService|GetService|BuildServiceProvider' -g '*.cs' rg -n 'AddEntityFramework|UseNexus|AddDbContext|SaveChanges|CommitAsync|\.Commit\(' -g '*.cs' rg -n 'ExternalPending|AddNexus.*Queues|EnqueueFrom|AddAdminUI|UseNexusAdminUI|AddApi|TeamsPublisher|OnInstanceEventRecieved|RequireTypeHash' -g '*.cs' ``` Record these decisions before changing behavior: | Area | What to establish | | -------------------- | ----------------------------------------------------------------------------------------------------------------------- | | Persistence | SQL Server, PostgreSQL, SQLite, in-memory fallback, and any custom storage implementations | | Queue triggering | Each queue job's actual saved schedule, whether it was manually started, and callers that explicitly started processing | | Job starts | Call frequency, immediate wake-up signals versus persisted requests, and how pending work is recovered after downtime | | Dependency injection | Singleton/root-provider callers of Nexus, and custom replacements registered with old lifetimes | | EF | Configured context, scope ownership, save/commit paths, and enqueueing that bypassed the unit of work | | Brokers | External pending storage versus ingress into Nexus; direct producers, old resources, backlog, offsets, and provisioning | | Serialization | Default STJ or Newtonsoft.Json, custom converters/writers, persisted polymorphic data, and the existing type-hash salt | | Clients | Custom HTTP/TypeScript clients, raw SQL, monitoring queries, and separate UI hosting | If saved schedules or broker state cannot be inspected locally, list the required environment checks in the deployment instructions. Do not assume code defaults describe an existing installation. Also establish the Nexus version actually deployed against each database and whether its startup/queue initialization completed. Package references in the current checkout are not evidence that the required v2 deployment has run. ### Questions to resolve with the user Inspect the application first, then ask about applicable behavior choices that the repository and existing instructions do not answer. Present actual job names, saved schedules where available, and a recommendation. Group related questions and continue mechanical API/runtime changes while awaiting answers. Do not ask the user to decide straightforward renames, repeat decisions already made, or silently treat an unanswered recommendation as accepted. | When applicable | What to ask and recommend | | --------------- | ------------------------- | | Queue jobs polling every five seconds or more often | List every queue job whose effective schedule runs at intervals of **five seconds or less**. Recommend converting **all of them** to queue jobs without schedules, so readiness starts processing. Ask which, if any, need an exception because the schedule enforces business hours, batching, downstream rate limits, or another business rule. | | Other scheduled or previously manual-only queue jobs | Ask which should retain scheduled processing and which should run when messages are ready. Preserve slower schedules unless a change is agreed. Explicitly ask how a previously manual-only job should behave: an empty schedule now enables readiness-driven processing. | | Choosing the processing model | Recommend retaining `IQueueJob`/`IBatchQueueJob` for the upgrade. Removing frequent polling does not require a distributed queue processor. Ask about adopting processors only if the user wants their different concurrency/execution model; first explain any affected job parameters, mutexes, progress, and run history. | | Explicit job-start calls | Establish call frequency and whether each call is a wake-up signal or a request that must itself be persisted. Recommend keeping frequent immediate calls on `IJobStartRequester`: v3 avoids extra-run database writes and the associated cache invalidations across instances. Do not convert frequent callers to scheduling extra runs as part of the upgrade. Ask about durable extra runs only for specific persisted-start requirements, explaining their additional cost and checking how the application already recovers pending work. | | EF callers whose transaction boundary is unclear | Ask whether the message must commit with the entity changes or be published independently. Recommend preserving the existing business transaction; do not choose immediate publishing merely to make a compile error disappear. | | Queue jobs with progress reporting | Explain that automatic progress now defaults to `None`. Ask whether users need percentages, processed counts, or no automatic progress. Recommend explicitly retaining a relied-upon display; explain the initial count query needed for percentages. | | Deployment history or broker cutover is unknown | Ask whether the latest v2 was deployed and initialized against every target database/queue, and what evidence is available. For incompatible broker changes, ask whether to drain or transfer existing backlog and who handles the cutover. Prepare the plan without claiming these environment steps have happened. | For the frequent-polling question, show a table with job name, queue name, environment, code default, actual saved schedule, proposed change, and any known restrictions. For six-field cron expressions, `* * * * * *` runs every second and `*/5 * * * * *` every five seconds: both qualify. Expand application schedule helpers and inspect effective saved schedules rather than matching expression text. Use the actual cron dialect and timezone when evaluating cadence. A restricted-hours schedule may poll every five seconds while active, but removing it also removes that restriction; make this consequence explicit. An example question, populated with the jobs actually found: > These queue jobs poll every five seconds or faster: [job names and effective schedules]. I recommend removing the schedule from all of them so they process when messages are ready. This also means clearing their saved schedules in each environment. Should any retain a schedule because they enforce processing windows, batching, or downstream limits? If saved schedules are inaccessible, identify candidates from code but mark their environment state unverified. Record accepted changes, exceptions, and unresolved choices. Without an answer, preserve the existing configured behavior where possible and leave the proposed behavior change pending; explicitly flag v3 defaults, such as an empty schedule becoming readiness-driven, that prevent automatic preservation. ## 2. Update the runtime and packages ### Required intermediate deployment For every existing installation, **deploy and run the latest Nexus v2 release before deploying v3**, even if the application is already on an earlier v2 release. The supported upgrade path requires this separate deployment so v2's database migrations and compatibility work complete before v3 adopts and caches the schema. 1. Verify the version actually running in each environment. If the latest v2 has not already been deployed successfully, prepare that deployment first. 2. Let startup migrations and queue initialization finish against every Nexus database being upgraded. When different hosts register different queue sets, run the relevant hosts against those databases so all queue initialization is covered. Resolve failed or incomplete initialization before proceeding. 3. Record the deployed v2 version and evidence of successful initialization in the deployment plan. Updating package references, compiling v2, or testing against a different database does not satisfy this prerequisite for the target environment. If the latest v2 is already deployed and fully initialized, no repeat deployment is needed. If deployment history cannot be verified, code preparation for v3 can continue, but report the prerequisite as unresolved and do not mark the v3 rollout ready. Fresh v3 installations with no existing Nexus data do not need an intermediate v2 deployment. ### Runtime and package changes All Nexus v3 packages target .NET 10. Update consuming projects to .NET 10 or later, and align the SDK, runtime/ASP.NET hosting bundle, containers, and CI. Upgrade the `CommerceMind.Nexus.*` packages together, including abstractions, storage providers, EF, UI, API client, serializers, and integration packages actually in use. Check transitive references from shared libraries too. V3 cannot upgrade v1 directly. An installation still on v1 must complete the same latest-v2 deployment first. V3 no longer adds v1 queue priority columns or rewrites old serialized type/assembly names. Inspect persisted queue messages, job parameters, function arguments, instance events, health-check results, and integration state when v1 data may remain. Successful schema initialization alone does not prove every old payload has been rewritten; process or migrate remaining v1 payloads through v2. Remove `NexusOptions`, `SkipV1Compatibility`, and the `AddNexus(Action)` overload: ```cs // v2 services.AddNexus(options => options.SkipV1Compatibility = true); // v3 services.AddNexus(); ``` The public `JsonTypeName.UpdateMovedTypeNames` helper was also removed. Do not recreate v1 compatibility inside the application's normal v3 startup. ## 3. Migrate job APIs without changing job identity Most schedule-neutral names lost the `Scheduled` prefix. Namespaces generally remain under `CommerceMind.Nexus.Abstractions.Jobs`, `.Jobs.Attributes`, and `.Jobs.Queueing`; registration remains under `CommerceMind.Nexus.Jobs.Extensions`. | V2 API | V3 API | | ------------------------------------------------------------------- | --------------------------------------------- | | `AddScheduledJobs()` | `AddJobs()` | | `ScheduledJobsOptions` | `JobsOptions` | | `ScheduledJobsOptions.ScheduledJobFilter` | `JobsOptions.JobFilter` | | `ScheduledJobsOptions.GeneratedScheduledJobs` | `JobsOptions.GeneratedJobs` | | `[ScheduledJob]` / `ScheduledJobAttribute` | `[Job]` / `JobAttribute` | | `IScheduledQueueJob` / `` | `IQueueJob` / `` | | `IScheduledBatchQueueJob` and its generic variants | `IBatchQueueJob` and its generic variants | | `IScheduledQueueJobBase` / `` | `IQueueJobBase` / `` | | `IScheduledQueueJobExecutor` | `IQueueJobExecutor` | | `GeneratedScheduledQueueJob` / `IGeneratedScheduledQueueJob` | `GeneratedQueueJob` / `IGeneratedQueueJob` | | `GeneratedScheduledQueueJob.ScheduledJobBaseType` | `GeneratedQueueJob.JobBaseType` | | `IScheduledJobMetaDataRepository` | `IJobMetaDataRepository` | | `IScheduledJobExecutor` / `IScheduledJobExecutorService` | `IJobExecutor` / `IJobExecutorService` | | `IScheduledJobMiddleware` | `IJobMiddleware` | | `IScheduledJobMetrics` / `IScheduledJobMetricsFactory` | `IJobMetrics` / `IJobMetricsFactory` | | `ScheduledJobDescriptor` / `ScheduledJobDescriptorExtensions` | `JobDescriptor` / `JobDescriptorExtensions` | | `ScheduledJobDescriptor.ScheduledJobType` | `JobDescriptor.JobType` | | `ScheduledJobMetaData`, `ScheduledJobCategory`, `ScheduledJobMutex` | `JobMetaData`, `JobCategory`, `JobMutex` | | `ScheduledJobCurrentRunDetails`, `ScheduledJobRunDetails` | `JobCurrentRunDetails`, `JobRunDetails` | | `ScheduledJobHistoricalRunDetails`, `ScheduledJobExtraRun` | `JobHistoricalRunDetails`, `JobExtraRun` | | `ScheduledJobRunInitiator`, `ScheduledJobRunOrder` | `JobRunInitiator`, `JobRunOrder` | | `ScheduledJobProgress`, `ScheduledJobHealthCheckData` | `JobProgressDetails`, `JobHealthCheckData` | Do not apply a global `ScheduledJob` to `Job` replacement. Ordinary cron jobs still use `IScheduledJob`, `IScheduledJob`, and `IScheduledJobBase`. `NextScheduledJobRun` also keeps its name. The common base for both ordinary scheduled jobs and queue jobs is now `IJobBase`; generic infrastructure intended for both should use it. Code that assumes every queue job is an `IScheduledJobBase` must be updated. Preserve explicit attribute/generated names. If a job was named by its class, keep that class name or set `[Job("existing-name")]` before renaming it. Job names identify schedules, history, parameters, and administrative state. Queue names identify storage. When renaming application classes, update matching filenames according to repository conventions. ### Removed members and repository changes - `JobAttribute` and `JobDescriptor` no longer have the old `Description` property. Use `ShortDescription` for the code-defined summary. This does not remove editable `JobMetaData.Description`. - `StopProcessingOnError` was removed from queue-job interfaces. Set `[Job(PauseOnError = true)]` or the generated job's `PauseOnError`. Check saved settings that may override the code default. - `IScheduledFilteredQueueJobBase`, `IScheduledFilteredQueueJob<...>`, and `IScheduledFilteredBatchQueueJob<...>` were removed. Preserve the filter through generated queue routing (`ShouldEnqueue`) or an explicit `GeneratedQueueJob.MessageFilter`, according to whether the filter selects the destination or messages to lease. The generated job filter is not derived from job parameters. Preserve each existing queue/job binding and verify its selected messages; do not silently replace filtered processing with an unfiltered job. - The unscoped `ScheduledJobInstance` property was removed. Use `JobDescriptor.ScopedJobInstance`, inspect its exception result, and dispose its returned scope after using the job. The old scoped member was named `ScopedScheduledJobInstance`. - Job metadata reads no longer load progress by default. Pass `loadJobProgress: true` to `GetJobsMetaDataAsync()` or the `GetJobMetaDataAsync()` overload when the caller needs it. - `TrySetAsRunningAsync()` takes `JobRunStart`, containing `StartedAtUtc`, `InstanceId`, `Initiator`, `Parameters`, and `CorrelationId`. `JobCurrentRunDetails` has an implicit conversion, but custom repository implementations must implement the new signature. - Historical-list queries take `JobHistoricalRunQuery` and return `JobHistoricalRunResult` with `RunDetails` and `TotalCount`. The result can still be deconstructed. Transfer all old filtering, ordering, and paging arguments into the query object. - Historical runs no longer expose a database `Id`. Use `CorrelationId`. - `JobExtraRun.Id`, extra-run deletion arguments, and HTTP extra-run IDs are `long`, not `int`. - Custom job repositories must implement both `GetMessageFilterAsync()` overloads; their default interface implementations were removed. Review the complete target interface for queue-job and nullable-schedule changes. ## 4. Preserve queue triggering and job-start guarantees ### Queue jobs An `IQueueJob`/`IBatchQueueJob` has no default schedule. A job with no saved schedule is readiness-driven: committed pending messages request activation, and activation starts processing. A job with a cron schedule is scheduled-only. Manual/extra runs remain available. Replace every `EnqueueAndStartProcessingAsync(...)` with the equivalent `EnqueueAsync(...)`, preserving context and destination. The old methods, `IQueueProcessingRequester`, and its implementation were removed. Advanced integrations use `IQueueReadinessNotifier` for queue readiness; do not replace normal enqueueing with a manual job-start event. Choose the intended behavior for each existing queue job: - To preserve scheduled-only processing, retain its saved cron schedule. Enqueueing does not bring that run forward. - To start on readiness, omit `DefaultSchedule` for new jobs and clear the persisted schedule on existing jobs through the Admin UI/API or `IJobMetaDataRepository.UpdateJobScheduleAsync(jobName, null)`. Changing the class default alone does not update an existing database row. - A previously manual-only queue job with an empty schedule now qualifies for readiness-driven processing. If it must remain manual, explicitly design that behavior using disabling/enabling or a suitable schedule and manual runs; do not assume an empty schedule still means manual-only. - If an old enqueue-and-start call intentionally woke a scheduled queue job before its next cron tick, enqueue and then use `IJobStartRequester` to request that run. Keep frequent wake-up calls on the fast requester path. Use a durable extra run only when the start request itself specifically needs persistence, rather than treating every enqueue as a reason to write an extra run. ### Convert frequent polling to readiness Recommend removing the schedule from **every queue job scheduled to run every five seconds or more often**, subject to the user's decisions above. This means a polling interval of **at most five seconds**, including exactly five seconds. It applies to queue-processing jobs, not ordinary scheduled jobs that perform periodic work without a queue. For each agreed conversion: 1. Keep its `IQueueJob`/`IBatchQueueJob` implementation, explicit job and queue names, message contracts, filters, and processing logic. Remove its `DefaultSchedule` override, or set the generated queue job's `Schedule` to null. Check shared base classes and registration helpers for inherited schedule defaults too. 2. Prepare an explicit list of the existing saved schedules to clear in each environment. Removing the code default alone does not change existing metadata. Use the Admin UI/API or the scoped `IJobMetaDataRepository.UpdateJobScheduleAsync(jobName, null)` during the coordinated v3 cutover, after the required latest-v2 deployment and v3 initialization. Clear only the jobs chosen for conversion. A null schedule enables readiness; it does not disable the job. 3. Preserve disabled states and pause-on-error settings. Verify the configured activation provider is shared by producers and workers, and that enqueueing requests readiness after commit. Do not retain a polling timer or add a manual start on every enqueue to emulate the old schedule. 4. Review `MaxNumberOfMessagesPerRun`, batching, and concurrency if the old run interval limited throughput. A job without a schedule can start again as work becomes ready; a per-run message limit does not impose the old time-based rate limit. If a business-hours window or downstream limit is required, agree how to retain it before removing the schedule. 5. Validate committed enqueueing, delayed messages, retries, an already populated queue at cutover, and idle behavior. Confirm the saved schedule is empty and new processing does not wait for a cron tick. Report both the code change and the per-environment metadata change, including any deferred execution. Do not automatically replace a five-second schedule with a five-second `ActivationDelay`. Activation delay is an optional fixed collection window with different semantics; choose it only when the user wants that behavior. ### Relevant v3 contract excerpts These are selected members from the v3 contracts, included for offline reference. They omit other members and are not replacement interfaces to add to the application. Types are under `CommerceMind.Nexus.Abstractions.Jobs`, `.Jobs.Queueing`, and `.Queueing`. ```cs // IQueueJobBase : IJobBase string? DefaultSchedule => null; QueueJobProgressMode AutomaticProgressMode => QueueJobProgressMode.None; int NumberOfProcessingThreads => 1; TimeSpan MessageLeaseTime => TimeSpan.FromMinutes(30); int MaxNumberOfMessagesPerRun => int.MaxValue; // IJobMetaDataRepository Task> GetJobsMetaDataAsync( bool bypassCache = false, bool loadJobProgress = false); Task UpdateJobScheduleAsync(string jobName, string? cronSchedule); Task ScheduleExtraRunAsync( string jobName, TimeSpan startIn, JobRunInitiator initiator, object? parameters = null, string? instanceId = null); Task ScheduleExtraRunAsync( string jobName, DateTime startAt, JobRunInitiator initiator, object? parameters = null, string? instanceId = null); ``` For example, inside an existing application scope, this reads saved metadata without the repository cache and clears one selected queue job's schedule: ```cs var jobs = await jobMetaDataRepository.GetJobsMetaDataAsync(bypassCache: true); // Inspect jobs' Name and Schedule; use the agreed existing job name below. await jobMetaDataRepository.UpdateJobScheduleAsync("existing-queue-job-name", null); ``` Prepare this as a deliberate cutover operation, not an unconditional startup loop that overwrites later user edits. The implementation normalizes null to an empty stored schedule; queue-job activation checks for an empty/whitespace schedule. Public metadata therefore need not return null after clearing it. ### Activation and progress Queue readiness and `IJobStartRequester` are different mechanisms. Queue activation retains coalesced readiness state; a job-start event does not. SQL Server, PostgreSQL, and SQLite providers register database activation storage automatically. `AddRedisQueueActivations(...)` selects Redis instead and is independent of `AddRedisExternalPendingStorage(...)`. `AddMemoryQueueActivations()` selects process-local activation state for a single-instance deployment and does not survive a restart. Use a consistent shared activation provider and namespace on producers and workers. Only one explicitly selected activation provider is supported. Activation is requested after the queue operation commits; do not assume an atomic enqueue-plus-activation transaction merely because both stores use SQL. A crash between commit and activation is recovered by reconciliation. `JobsOptions.QueueActivationReconciliationInterval` defaults to ten minutes, and `QueueActivationFailureDelay` defaults to five seconds. Reconciliation is a correctness fallback, not a substitute for correct producer readiness. `ActivationDelay` is a fixed burst-collection window, not a sliding debounce. Additional enqueues do not extend the pending deadline. `ProcessingDelay`/`ProcessAtUtc` determines when each message is eligible. Configure activation delay on the queue attribute/named or generated definition, or through queue metadata/API. Automatic queue-job progress defaults to `QueueJobProgressMode.None`. Set this property to retain percentages: ```cs public QueueJobProgressMode AutomaticProgressMode => QueueJobProgressMode.Percentage; ``` `ProcessedCount` avoids the initial pending-count query. All automatic modes aggregate updates per lease/batch; injected `IJobProgress` still works. External queue counts may be approximate and may require broker-management permissions. ### Explicit job starts **Keep frequent immediate job-start calls on `IJobStartRequester`.** The v3 implementation deliberately makes these calls faster by broadcasting an instance event instead of writing an extra run to the database. This also avoids invalidating the extra-run cache on every instance when an extra run is scheduled and completed. Replacing frequent calls with `ScheduleExtraRunAsync()` would give up that improvement; do not make that replacement as a routine upgrade step. `RequestJobStartAsync(jobName)` and the parameterless generic overload use this fast path. Calls whose actual parameters and start time are both null also take it, even if a parameter-taking overload was used. The event is a best-effort wake-up signal: requests can be lost during downtime, coalesced, or cause successive runs on different instances. A request received while the job is running can trigger another run afterward, so it can pick up work that arrived near the end of the current run. The job mutex still prevents concurrent runs of the same job; one request does not guarantee exactly one run. Ask about the caller's purpose and frequency before changing its execution model. Distinguish a signal to process work already stored elsewhere from a start request that is itself the only record of required work. Check the application's existing recovery path for missed signals, such as a later request or a periodic job run. A job having no schedule is a reason to inspect recovery, not an automatic reason to replace its frequent start calls with database writes. If a frequent caller needs stronger recovery, discuss recovery of its persisted work while retaining fast wake-up signals rather than automatically persisting an extra run per call. Only continuous job loops handle the event. Use `IJobMetaDataRepository.ScheduleExtraRunAsync()` when a particular start request must be persisted, or when the host relies on `RunAllJobsPendingSinceLastCallAsync()` and therefore cannot handle these events. Explain the persistence/performance tradeoff and limit the change to those specific callers: ```cs await jobMetaDataRepository.ScheduleExtraRunAsync( "existing-job-name", TimeSpan.Zero, JobRunInitiator.Manual); ``` Requests with a non-null parameter object or an explicit delay/start time still persist extra runs and require `AddJobs()` plus job-database access. Preserve these semantics where needed. Do not add dummy parameters, `TimeSpan.Zero`, or an explicit current timestamp to frequent immediate calls merely to force persistence; that bypasses the fast path. Do not rely on passing null to make a request durable. An event-only producer can use `IJobStartRequester` without `AddJobs()` or a local job registration when it configures the shared instance-event broker. It must use the correct target name. Generic requests fall back to `[Job("name")]` or the class name; multiple local bindings require an explicit name. Generated/per-queue names may differ from the class name. ### Schedule timezone `JobsOptions.ScheduleTimeZone` is a `TimeZoneInfo` and defaults to UTC. Preserve UTC during a behavior-preserving upgrade. To intentionally use local wall-clock schedules: ```cs nexus.AddJobs(options => { options.ScheduleTimeZone = TimeZoneInfo.FindSystemTimeZoneById("Europe/Stockholm"); }); ``` This affects all cron schedules, including ones saved through the UI/API. Configure the same effective timezone on every processing instance. `TimeZoneInfo.Local` depends on the server configuration; named IDs must resolve on the target host. Timestamps and extra-run dates remain UTC. Verify daylight-saving boundaries using the actual cron expressions; local daily schedules follow Cronos's gap/overlap handling. Do not convert stored UTC timestamps to local time. ## 5. Repair dependency injection lifetimes Operational Nexus services are generally scoped in v3: enqueuers/readers/updaters, repositories, functions, job execution, serializers, health-check implementations, and integration operations. Process-wide state such as `INexusInstance`, `IInstanceEventBroker`, `IGracefulShutdownService`, and `IDateTime` remains singleton. Inspect actual registrations for custom overrides rather than assuming every Nexus type has changed. Find singleton constructors, hosted services, cached delegates, static holders, and root-provider resolutions that capture scoped Nexus services, including indirect captures through transient dependencies. Either move an operation into an appropriate existing scope or create a scope for the independent operation: ```cs public class QueuePublisher(IServiceScopeFactory scopeFactory) { public async Task PublishAsync(MyMessage message) { await using var scope = scopeFactory.CreateAsyncScope(); var enqueuer = scope.ServiceProvider.GetRequiredService>(); await enqueuer.EnqueueAsync(message); } } ``` This example assumes enqueueing should persist immediately without EF interception. With EF enabled, apply the unit-of-work rules below. Keep scopes alive until asynchronous work finishes; do not return a scoped service or fire-and-forget its work. Controllers already have a request scope. Nexus job/function execution supplies a scope too. Do not make every application service scoped automatically. Keep singleton state where required and create scopes around its independent operations. Do not suppress DI validation to hide a lifetime error. Validate the application's service graph with scope validation enabled in a local/test host. Nexus's scoped hosted-service wrapper retains a scope for the hosted worker's lifetime; it does not make every scoped dependency a per-message dependency. Queue-job processing threads can still share the same job instance. Preserve thread safety and never run concurrent EF operations on one context. Distributed processors instead create a fresh scope and processor instance for each leased chunk. ## 6. Preserve Entity Framework transaction boundaries Apply this section only when the application uses `CommerceMind.Nexus.EntityFramework`. The integration no longer finds contexts through global diagnostic-event tracking. It resolves the one configured context type from the enqueuer's DI scope. Register the integration once, register that context as scoped, avoid `AddDbContextPool`, and use the scoped provider: ```cs services.AddNexus().AddEntityFramework(); services.AddDbContext((serviceProvider, options) => { options.UseSqlServer(connectionString); options.UseNexus(serviceProvider); }); ``` Database enqueue operations through `IEnqueuer`, `IEnqueuer`, `IDatabaseEnqueuer`, and `IObjectEnqueuer` now join the same scoped unit of work. In-memory queue additions happen after commit. Inspect all enqueue sites, including non-generic/batch/virtual/generated paths, for code that previously relied on a bypass. | Situation | V3 behavior / required action | | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------- | | Application updates entities and enqueues | Resolve context and enqueuer in the same scope; call `SaveChangesAsync()` | | Explicit EF transaction | Save and commit asynchronously; readiness is requested after commit | | Context disposed without save/commit | Pending database/in-memory enqueue operations are discarded | | Save fails | Pending messages remain with pending entity changes for a later successful save | | Save succeeds, transaction later rolls back | Queue operations from that transaction are discarded and are not replayed by a later save | | Synchronous save/commit with pending operations | Rejected; change to async throughout the call chain | | Remote API or external-pending enqueue | Sent immediately; cannot roll back with EF, even for external messages temporarily stored in SQL | | Queue updater/other Nexus operations | Not automatically enrolled by enqueue interception; use an explicit supported transaction when needed | A singleton's new scope cannot join an arbitrary caller's context. Move atomic entity-plus-enqueue work into a scoped application operation that owns both. Do not use an immediate-operation scope to work around that requirement. For independent infrastructure publishing that must persist before returning without saving EF, use: ```cs await using var scope = scopeFactory.CreateAsyncScope(); using var immediate = new ImmediateQueueOperationScope(); var enqueuer = scope.ServiceProvider.GetRequiredService>(); await enqueuer.EnqueueAsync(message); ``` `ImmediateQueueOperationScope` is in `CommerceMind.Nexus.Abstractions.Queueing`. Nexus already uses it for API/MCP enqueue endpoints, broker ingress, and hybrid-storage transitions. Use it deliberately for standalone operations; normal application writes should join EF. When remote/external delivery must follow an entity transaction, enqueue into a database queue as an outbox and forward after commit. Make forwarding safe to retry; an external publish and local acknowledgement are not one atomic transaction. ## 7. Migrate queue operations and custom implementations ### Batch request records and reads Tuple-based queue operations were replaced with records in `CommerceMind.Nexus.Abstractions.Queueing`: | Operation | Request type | | -------------------- | --------------------------------------------------------------------------------------------------------- | | Enqueue with context | `EnqueueRequest(Message, Context)` | | Delete by ID/version | `QueueItemDeleteRequest(Id, Version)` | | Mark processed | `ProcessedQueueItemUpdate(Item, ProcessingLog, ProcessAgainAtUtc, NewPriority, UpdatedMessage)` | | Mark failed | `FailedQueueItemUpdate(Item, Error, RetryAfter, StatusWhenMaxRetried, UpdatedMessage)` | | Set status | `QueueItemStatusUpdate(Item, Status, ProcessingLog, UpdatedMessage)` | | Release lease | `QueueItemRelease(Item, NewPriority, UpdatedMessage)` | For example: ```cs await enqueuer.EnqueueAsync(messages.Select(message => new EnqueueRequest(message, new EnqueueContext { ProcessAfter = TimeSpan.FromMinutes(5) }))); ``` Single-message enqueue methods now accept `EnqueueContext?`, typically defaulting to null. `IObjectEnqueuer` batch requests use `EnqueueRequest`. Custom enqueuers must implement the new request-based methods and named-destination methods required by their interface. The non-generic `IQueueReader.ReadAsync()` now puts `filter` before sort/paging arguments and returns `QueueReadResult` (`QueueItems`, `TotalCount`, still deconstructable). Revisit positional calls. Typed reader overloads still return lists; do not rewrite those as though they return `QueueReadResult`. ### Options and storage members - `QueueAttribute.KeepProcessedMessagesFor` and `QueueSystemOptions.KeepProcessedMessagesFor` become `ProcessedMessagesRetention`. Update named attribute arguments such as `[Queue(ProcessedMessagesRetention = "10")]` too; the attribute value remains a duration string. - `QueueDescriptor.KeepProcessedMessagesFor` was removed; use `ProcessedMessagesRetention`. - Obsolete `InMemory` properties on attributes, generated queues, descriptors, and `IGeneratedQueue` were removed. Use `StorageMode = QueueStorageMode.InMemory`. - `IQueueReader.QueueDisplayName` was removed; use the queue descriptor's `DisplayName`. - `IQueueStorage.QueueDisplayName`, `KeepProcessedMessagesFor`, `MaxRetries`, and `HistoryRetention` were removed. Read descriptor defaults from `Descriptor` or current settings from `IQueueMetaDataRepository`. - Table names moved to `IDatabaseQueueStorage`: custom database implementations must supply `TableName`, `HistoryTableName`, and `StatisticsTableName`. - Statistics reader/writer methods now take `IDatabaseQueueStorage`. - `QueueItem.MessageObject`, `MessageObjectOrThrow`, and `PreviousMessageObject` are `IQueueMessage`, not `object`. - Custom `IQueueItemUpdater` implementations must supply the previously defaulted ID/version delete methods, typed release methods, and new batch methods. - Custom queue metadata repositories need `SetActivationDelayAsync()`. Implement saved processing settings too if runtime overrides are supported; the interface's default setter throws. - `QueueSystemOptions.DisableHistory` now wins over saved metadata. Review deployments that set it while expecting history enabled through the Admin UI; history tables/data may be removed during initialization. ### `AwaitingRelease` and custom queue processing For database and in-memory messages with identity, enqueueing a new version while leased now sets `AwaitingRelease` and retains the original lease. The new version cannot be leased concurrently. The latest enqueue determines the deferred payload, priority, processing date, and `StatusAfterRelease`; subsequent enqueues replace those deferred values. When the original processor completes or releases, apply the latest enqueue's destination. Its payload/priority/rescheduling/failure updates must not overwrite the newer enqueue. Successful processing still records the processed version as the previous message, even with zero processed-message retention. Expired leases follow normal abandonment and `RetryAbandonedMessages` rules. External pending storage retains provider-specific behavior and does not gain this protection. Update custom status switches, count displays, dashboards, raw SQL, serializers, and custom storage/updater implementations. Treat `AwaitingRelease` as an active lease, not ordinary pending work. `QueueItem.StatusAfterRelease` exposes its deferred destination. Do not force it to `Pending` merely to make an old status switch compile. ### Custom committers and external providers `IQueueOperationCommitter.AddOperationAsync()` now accepts a `QueueOperation` record. Its `ExecuteAsync` performs database work; `AfterCommitAsync` must run for every committed operation; `GetReadyQueueName` is evaluated after that callback succeeds. Coalesce readiness per queue only within the committed unit of work. Rolled-back operations must not publish readiness. In-memory callbacks must finish before consumers are woken. Custom external-pending providers should supply an `IExternalQueueReadinessBridge` that durably observes readiness without leasing/settling processing messages, and acknowledges notifications only after activation succeeds. `IExternalPendingStorage.MarkProcessedForIdempotencyBatchAsync()` has a default per-message fallback; override it for native batching if useful. Review queue-aware `IQueueMetricsFactory` overloads if custom metrics must distinguish named queues or distributed processors. ## 8. Update virtual queues and broker ingress Both sync and async virtual transformers must accept `EnqueueContext?` and return `TransformResult`. The obsolete one-argument methods were removed: ```cs public TransformResult Transform(SourceMessage message, EnqueueContext? context) { return new(new ExportMessage { Id = message.Id }, context); } ``` Returning a result with no replacement context inherits the incoming context. A null transformed message/empty destinations filters it out. Explicit queue destinations and `TransformResult.ToMany(...)` are available when needed. Fan-out to several destinations does not create an atomic cross-provider transaction, and partial delivery can repeat on retry. Custom implementations of the source-only transformer interface must implement `TransformDestinationsAsync()`. Kafka/RabbitMQ ingress registrations now distinguish the wire type from the Nexus payload type. Removed overloads accepted `Func` or a runtime `Type`. Use the typed overload, with the Nexus type first: ```cs kafkaQueues.EnqueueFrom( "source-topic", message => new MyNexusMessage { Id = message.Id }); rabbitQueues.EnqueueFrom( sourceQueue, message => new MyNexusMessage { Id = message.Id }); ``` If raw text is required, explicitly choose the wire type/deserializer appropriate to the payload; do not assume the old string lambda still receives raw JSON. `IKafkaQueueMessageDeserializer.Deserialize()` and `IRabbitMqQueueMessageDeserializer.Deserialize()` return `object?`. Custom Azure deserializers must implement `DeserializeArray()`; the old default implementation is gone. Ingress registrations can take `nexusQueueName` to choose a concrete or virtual destination. Keep existing source subscriptions/queues unless the upgrade explicitly requires changing them. This ingress API is separate from Nexus external pending storage. ## 9. Plan external pending storage cutover Apply this section for `QueueStorageMode.ExternalPending`. Database migration does not move broker messages or create readiness for old resources. Inventory direct producers and backlog before cutover. Prefer draining through v2 when practical; otherwise prepare an explicit data-transfer plan preserving message identity, payload, delays, priority, and provider state. Do not delete old resources until the backlog and any required idempotency state have been accounted for. | Provider | V3 processing/readiness contract | Upgrade action | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Redis | Ready set `{KeyPrefix}{queue-name}`, delayed/ID/idempotency keys in the same hash slot, plus a readiness stream. Literal braces surround the logical queue name. With defaults: `nexus:queue:{orders}` and `nexus:queue:{orders}:readiness`. | Old unbraced keys are different resources. Drain or migrate ready/delayed messages and needed identity/idempotency state. Direct writers must atomically add ready messages and readiness records. | | RabbitMQ | Durable fan-out exchange `{ExchangeNamePrefix}{queue-name}` bound to processing and readiness queues. Defaults for `orders`: exchange `nexus:external-pending:orders`, processing queue `nexus:orders`, readiness queue `nexus:orders:readiness`. | Publish to the exchange. Existing messages in the processing queue have no new readiness copy; drain or deliberately wake/reconcile them. Update direct writers and permissions for declarations/bindings. | | Kafka | Same topic, independent processing and readiness consumer groups. Default topic: `nexus-orders`. | Configure distinct group IDs and retain the intended processing offsets. Readiness commits must never advance processing offsets. Grant access to both groups. | | Azure Service Bus | Topic `{QueueNamePrefix}{queue-name}` with processing and readiness subscriptions. Defaults: topic `nexus-orders`, subscriptions `processing` and `readiness`. | Old queue entities are not consumed by v3. Drain or migrate them and switch writers to the topic. Provision both subscriptions before sending. Nexus needs Manage permission if it creates topology. | Redis's literal cluster hash tag must match on all related keys. Do not assume `RENAME` works across Redis Cluster slots, and do not migrate only the ready set while ignoring delayed/ID/idempotency structures. The provider's idempotency sorted-set suffix is `:idempotent`; its full key also gains the braces in v3. Inspect provider code when transferring those records. Broker bodies contain the Nexus external-pending envelope, not merely the application payload: ```json { "MessageJson": "{\"OrderId\":\"123\"}", "MessageId": "123", "Priority": 0 } ``` `MessageJson` is serialized payload JSON inside the envelope string. See [external pending storage](../queues/external-pending.md#readiness-bridges-and-direct-producers) for the direct Redis enqueue script and provider contracts. Nexus `IEnqueuer` already writes the v3 topology correctly. Redis direct producers needing full ID replacement/idempotency should use it rather than a minimal sorted-set script. Provider bridges acknowledge their notification only after durable activation; they do not settle the processing copy. Check the `NexusExternalQueueReadiness_{ProviderId}` health check. Reconciliation can recover missed notification signals for the current processing resource, but cannot discover a message left in an old Redis key or Service Bus queue entity. ## 10. Consider optional processors, named queues, and registration changes Do not convert existing jobs automatically. When requested, distributed `IQueueProcessor`/`IBatchQueueProcessor` implementations live in `CommerceMind.Nexus.Abstractions.Queueing.Processing` and register through `AddQueueProcessors()` after `AddQueues()` (and after `AddJobs()` when both are used). Distributed processors run `ConcurrencyPerInstance` workers on every instance. They use atomic leases, are at-least-once, and provide no ordering across workers/instances. Set lease time above the maximum expected processing time of the whole chunk. Single-message processors default to `LeaseSize = 1`; batch processors default to lease size 50, capped by batch size. Each leased chunk has a new scope/processor instance invoked sequentially. This is different from a multithreaded queue job sharing its job instance. Processors have queue-level pause/status controls, not job runs, schedules, parameters, mutexes, or job progress. In-memory queues are unsupported. A concrete queue cannot have both a queue job and distributed processor. Account for total concurrency across queues and hosts before opting in. Named queues (`NamedQueue` in `QueueSystemOptions.NamedQueues`) explicitly share one payload type. Adding a named definition suppresses automatic ordinary concrete discovery for that type. Enqueue by name even if only one named queue exists; unqualified typed DI cannot select a named destination. Use `IQueueResolver.GetEnqueuer(name)`, `GetReader(name)`, or `GetUpdater(name)` for a bound accessor. `IApiEnqueuer` also supports named destinations without local storage registration. One ordinary processing implementation binds to every matching concrete queue. Named job bindings default to `{job-name}_{queue-name}`; existing single ordinary queue jobs retain their names. Use explicit generated job definitions to preserve previous per-queue job names when changing registration. Single-message handlers can inspect `QueueProcessingContext.QueueName`; batches expose `QueueName` and `Context`. ### Existing generated queues require migration This is required for applications already using generated queues, even if they do not adopt any additional named queues. `GeneratedQueue` now inherits the named queue model. Nexus no longer emits derived message or job types. Make abstract message contracts and generated job implementations concrete; open generic job implementations can still be closed over the original message type. Remove dependencies on generated runtime type names or casts to emitted subclasses. Preserve ordinary serialized payload compatibility and verify old messages remain readable. If several old filtered jobs shared one physical queue, v3's one-processing-job-per-queue rule requires an explicit redesign; do not discard or redistribute its existing backlog implicitly. Keep each generated queue's existing `Name` and each generated job's explicit `Name` and `QueueName`. Rename `ScheduledJobBaseType` to `JobBaseType` and `GeneratedScheduledJobs` to `GeneratedJobs`. Set a shared `RoutingQueueName` on generated definitions whose `ShouldEnqueue` predicates previously selected destinations. Send messages to that routing name: ```cs new GeneratedQueue { Name = "language_sv", DisplayName = "Swedish", RoutingQueueName = "languages", ShouldEnqueue = message => message.Language == "sv" }; // Evaluates ShouldEnqueue for destinations sharing this entry point: await enqueuer.EnqueueAsync("languages", message); // Goes directly to this queue, without evaluating the routing predicate: await enqueuer.EnqueueAsync("language_sv", message); ``` Type-only enqueueing no longer broadcasts to generated queues or evaluates `ShouldEnqueue`. Audit producers, including API/broker ingress and virtual transformers. Preserve the distinction between a routing predicate evaluated at enqueue time and a generated job's `MessageFilter` evaluated when leasing. See [generated queues](../queues/generated.md) for a full registration example. Keep names stable and unique ignoring case. Concrete storage names use letters, digits, and underscores; virtual names can also contain hyphens/periods. Nexus validates backing-table collisions, including provider truncation. Message identity/history/idempotency are independent in each queue, while validators and priority calculators remain type-based. Repeated `AddQueues()`/`AddJobs()` calls share options and collect contributions. Append with `Add`/`AddRange`; assigning a new list replaces previous libraries' entries. Scalars use the last value. Each scan captures its filter at that call; duplicate discoveries do not duplicate registration. `Build()` registers all queues before jobs/processors and rejects duplicate names and conflicting bindings. Call it after all contributions. Generated job definitions still require their queue declarations before that `AddJobs()` call. `AddQueues()` alone does not enable processing. Runtime processing overrides are saved per queue. For jobs they include threads, lease time, and batch size; distributed processors also expose worker count and lease size. Null override fields use code defaults; posting `{}` to `/api/queues/{queueName}/processing-settings` resets overrides. Active jobs keep their current snapshot until the next run. Distributed pools finish current work then rebuild; existing lease expiries do not change. Read `defaults`, `overrides`, `effective`, and `pendingChanges` when diagnosing why code changes have not taken effect. The legacy UI does not expose these controls. ### Queue maintenance is independent of the job loop `AddQueues()` now registers its own maintenance worker. Readiness maintenance promotes due sleeping/retrying messages and recovers expired leases; cleanup deletes retained messages/history/statistics in bounded batches. Reducing `JobsOptions.JobLoopDelay` no longer controls this work. Custom hosts that invoke maintenance themselves can still call `OrganizeMessagesAsync(force: true)`; the compatibility entry point uses the split implementation. `QueueSystemOptions` defaults are `MaintenanceFallbackInterval = 10s`, `CleanupInterval = 1m`, `CleanupBatchSize = 1000`, and `MaxCleanupBatchesPerRun = 10`. The process wakes for known retry/lease deadlines; the fallback scan catches work/deadlines from other instances or restored after restart. Larger fallback intervals reduce idle database work but can delay that discovery. Cleanup limits apply per retention store per queue, so a large backlog may need several passes. Review timing assumptions in maintenance monitoring rather than changing these defaults automatically. ## 11. Separate Admin UI registration and update clients `CommerceMind.Nexus.Api` no longer brings in or registers `CommerceMind.Nexus.UI`. Add the UI package only to hosts that serve it. Move options to `AddAdminUI()` and use `CommerceMind.Nexus.UI.Extensions` for registration/middleware: ```cs services.AddNexus() .AddApi(options => options.BasePath = "/nexus/") .AddAdminUI(options => { options.ApiBaseUrl = "/nexus/"; options.EnvironmentName = environmentName; options.InfoNotice = "Test environment"; }) .Build(); // After building the ASP.NET application: app.UseNexusAdminUI(); ``` `ApiBaseUrl` is required. When colocated, match `NexusApiOptions.BasePath`. `NexusApiOptions.EnvironmentName` and `.AdminUI` were removed. `NexusAdminUIOptions`, `NexusAdminUIMenuItems`, and `IAdminUIFileProvider` moved to `CommerceMind.Nexus.UI`. `UseExperimentalNewVersion` was replaced by `UseLegacyVersion`, default false. Remove a true experimental flag to keep using the new UI; set `UseLegacyVersion = true` only when the legacy UI is intentionally needed temporarily. The default standalone UI path is now `nexus`; explicitly configure another `Path` if required. API base URL and UI path are separate settings. Check redirects, proxy path bases, links, authentication, and cross-origin access for separate hosting. In the final v3 implementation, `GET /api/enabled-features` is served by the API and describes API-host capabilities, even for a remote UI. Menu customization comes from `NexusAdminUIOptions.MenuItemsLoader` on the UI host. Do not move feature discovery to the UI host based on earlier prerelease guidance or depend on removed menu fields in the feature response. Regenerate custom clients from the actual v3 host's OpenAPI or update them explicitly. Routes below are relative to the API base path: | Contract | V3 change | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | `GET /api/jobs` | `{ "jobs": [...], "categories": [...] }` instead of an array | | `GET /api/jobs/categories` | Removed; use categories in the jobs-list response | | `GET /api/queues` | No counts by default. Use `queueNamesWithCounts` or `includeCountsForAll=true`; `defaultWithoutCounts` was removed | | Historical job runs | No database `id`; key by `correlationId` | | Extra runs / `nextRun.extraRunId` | 64-bit IDs; remove 32-bit casts/validators | | Queue create payloads | Removed `startProcessing`; activation follows the processing mode. Single-item payloads support `partialMessage` | | Job/queue responses | Nullable queue-job schedules, activation state/delay, processor data, `AwaitingRelease`, and `statusAfterRelease` | Preserve 64-bit IDs in client languages/storage that cannot represent every `long` exactly. Treat omitted counts as not loaded, not zero. Review exhaustive enum/status handling and code assuming every job has a next scheduled run. ## 12. Serialization, events, functions, and Teams `JsonOptions.RequireTypeHash` now defaults to true for System.Text.Json polymorphic deserialization. JSON containing `$type` must carry a valid `$typehash`. The default writer already writes hashes, so do not change normal payloads just because the default changed. This is not a requirement to put `$type` on every ordinary queue payload. The built-in STJ and Newtonsoft.Json health-result readers recognize v2's `ScheduledJobHealthCheckData` in cached results and historical statistics. They preserve stored payloads and validate STJ hashes against the original type name and salt; newly saved results use `JobHealthCheckData`. Custom health-result readers must retain equivalent compatibility. Rollback preserves original v2 rows and archives incompatible v3 results before refreshing cached checks. Exercise representative persisted polymorphic messages, parameters, arguments, health results, and custom integration state with the selected serializer. Pay attention to historical v1/Newtonsoft-compatible data and custom writers. If hashes are absent, migrate/drain that data first, or temporarily configure: ```cs services.AddNexus().ConfigureJsonOptions(options => { options.RequireTypeHash = false; }); ``` Keep `WriteTypeHash` enabled for new STJ data and retain the existing `TypeHashSalt`. Custom writers must put `$typehash` before `$type`, and both before ordinary object properties, as the Nexus converter reads metadata in that order. A mismatched/invalid hash is not repaired by manufacturing hashes in stored JSON. Track the temporary compatibility setting and enable validation once old data is gone. Do not switch between STJ and Newtonsoft.Json as an incidental part of this upgrade; `RequireTypeHash` describes the STJ converter's behavior. `IInstanceEventBroker.OnInstanceEventRecieved` was removed. Replace event handlers with asynchronous subscriptions, keep their handle, and dispose it with the subscriber: ```cs private readonly IInstanceEventSubscription _subscription; // In the constructor: _subscription = instanceEventBroker.Subscribe(HandleInstanceEventAsync); // HandleInstanceEventAsync returns Task. public void Dispose() => _subscription.Dispose(); ``` `DbConnectionScope` moved from `CommerceMind.Nexus.Core.Database` to `CommerceMind.Nexus.Abstractions.Core.Database`. Preserve its transaction/suppression semantics while changing the using directive. `NexusFunctionServiceMethod.Service` was removed, and its constructor no longer takes `IServiceProvider`. Custom function invocation code must resolve `ServiceType` from the active scope. Do not retain a resolved application service globally. The old Teams webhook API (`AddTeamsPublisher()`, `NexusTeamsOptions`) was removed. Direct bot mode uses `AddTeamsBotPublisher()`/`NexusTeamsBotOptions` and `UseNexusTeamsBot()`; outgoing-only bridge mode is another option when the application is not publicly reachable. Keep this migration separate from generic HTTP webhooks. See [Teams integration](../healthchecks/teams.md) for credentials, endpoint setup, linking, and bridge configuration; old webhook credentials are not bot credentials. ## 13. Prepare database initialization and deployment First verify the [required latest-v2 deployment](#required-intermediate-deployment) has completed initialization against the target database. Then stop all v2 processes that share it before v3 initialization, including API/producer hosts that initialize Nexus. A mixed v2/v3 rolling deployment is unsupported: v2 uses old job tables and does not participate in v3's startup locks. Coordinate producer pause/drain and broker changes before cutover. ### Automatic schema migration Nexus migrates job tables while retaining their data. Update application-owned SQL, views, grants, reporting, monitoring, and maintenance scripts referencing the old names: | SQL Server v2 → v3 | PostgreSQL / SQLite v2 → v3 | | -------------------------------------------------- | --------------------------------------------------------- | | `NexusScheduledJobs` → `NexusJobs` | `nexus_scheduled_jobs` → `nexus_jobs` | | `NexusScheduledJobRuns` → `NexusJobRuns` | `nexus_scheduled_job_runs` → `nexus_job_runs` | | `NexusScheduledJobProgress` → `NexusJobProgress` | `nexus_scheduled_job_progress` → `nexus_job_progress` | | `NexusScheduledJobMutexes` → `NexusJobMutexes` | `nexus_scheduled_job_mutexes` → `nexus_job_mutexes` | | `NexusScheduledJobLogs` → `NexusJobLogs` | `nexus_scheduled_job_logs` → `nexus_job_logs` | | `NexusScheduledJobExtraRuns` → `NexusJobExtraRuns` | `nexus_scheduled_job_extra_runs` → `nexus_job_extra_runs` | Do not manually recreate tables or discard the migration ledger. Renamed migrations recognize previous names through `PreviousMigrationNameAttribute`. New migrations add queue activation, activation delay, processing settings, and job-category storage as needed. Lease indexes are rebuilt to match `priority DESC, id ASC`. PostgreSQL uses `CREATE INDEX CONCURRENTLY`; SQL Server builds a replacement nonclustered index while retaining the legacy index, then swaps names in a short transaction. These still need extra space, I/O, and on SQL Server transaction-log capacity. They do not promise zero blocking. Interrupted builds retain the old index until the replacement is valid and retry/resume on startup. SQLite's transactional rebuild blocks other writers; plan a maintenance window for large queues. ### Cached initialization and filter backfills Reserved `Nexus.Startup.*` key-value records store the schema revision, completed migration manifests, and per-queue schema definitions. On unchanged starts Nexus skips regular migration-history queries and queue table/index/column inspection. Package version changes alone do not invalidate these records. The first v3 startup reads the existing ledger rather than blindly replaying migrations. Each unstamped v2 queue is adopted separately, including queues first registered by another application later. Compatible filter columns/list tables retain their values. Missing/incompatible filter definitions are rebuilt from persisted payloads. New queues initialize with the current schema even when regular migrations have already completed. Changes to filterable property definitions, history/statistics settings, and message-ID uniqueness are checked independently of the migration manifest. Keep old payloads deserializable when filter backfills require reading them. Pending backfill plans are persisted before schema edits so initialization can resume after interruption. Coordinate old/new application instances around incompatible schema changes. Nexus does not inspect or repair unchanged tables after manual edits. Do not delete/edit startup markers as a routine repair strategy. Diagnose schema drift and prepare an explicit supported migration. Application-defined regular migrations participate in the manifest; user-defined idempotent migrations still run on every startup and should remain safe to repeat. SQL Server/PostgreSQL serialize v3 initialization with session-owned locks on dedicated connections opened through the configured connection manager. Custom connection openers remain responsible for authentication and connection initialization. Allow pool capacity for the lock connection alongside migration connections. Failed lock acquisition/release discards the session from its pool. SQLite does not coordinate concurrent initializers; use one process. Completion state is published after successful initialization. A recent foreign v2 migration claim blocks startup rather than being stolen; verify the old owner has stopped before any force-migration recovery. Interrupted v3 claims can be retried after acquiring the lock. Regular migrations must still tolerate interrupted execution/retry. If maintaining a custom Nexus database provider, inspect the v3 provider contracts and migrations: update both fresh-schema definitions and regular migrations, retain old migration identities, and support the initialization state/locking protocol. Fixed-schema changes belong in regular migrations plus the new-table baseline, not repeated per-queue compatibility work. ### Deployment sequence 1. Deploy the latest v2 to each existing installation and wait for successful migrations and initialization of all relevant queues. If already completed, verify and record that evidence. Do not skip this deployment when moving from an older v2 release. 2. Build and validate the upgraded code. Validate migration in a test environment with initialized v2 data and isolated broker resources. 3. Record existing job names, schedules, disabled states, parameters, pending extra runs, queue backlog, and relevant broker offsets/resources. Confirm runtime and migration permissions/capacity. 4. Pause producers as needed, drain or transfer incompatible broker resources, and stop every v2 instance. PostgreSQL/SQL Server capture their rollback checkpoint internally on the first v3 migration startup; there is no separate capture script to run. 5. Provision the v3 broker topology and coordinate direct-producer changes. Initialize the database with the v3 application; verify automatic checkpoint capture on PostgreSQL/SQL Server and wait for successful migration/backfill completion before routing normal traffic. Use one initializer for SQLite. 6. Verify preserved metadata, apply the agreed saved-schedule changes (including frequent-polling conversions) and processing overrides, then start v3 workers and resume producers. If the cutover tool requires a running host, keep normal processing paused while applying changes. Confirm readiness health and actual processing for each provider. 7. Monitor errors, backlog, retries, lease recovery, EF persistence, and resource load. Retain old broker resources while they are still needed for backlog processing or a planned return to v2. If returning to v2, use the rollback instructions to reverse the database changes and account for data written since cutover and any broker changes before starting the v2 application. ### Optional rollback to v2 Use this section if the user wants to test returning to v2 or include rollback in their deployment plan. The [PostgreSQL script](/upgrade/v3-to-v2-postgres.sql) and [SQL Server script](/upgrade/v3-to-v2-sqlserver.sql) only perform rollback of the built-in v3 database changes, consuming the checkpoint Nexus creates internally. Read [the rollback instructions](rollback-v3.md) and the actual selected script before planning its execution. These downloads do not require GitHub access. If unavailable in the LLM's environment, obtain the matching file from the user rather than inventing an inverse from table names alone. - Nexus automatically captures the initialized v2 baseline under the startup lock, before idempotent queue initialization and regular migrations. Record the database, explicit schema, checkpoint table, and application artifacts for the rollback. A capture error stops the upgrade before those schema changes proceed. - The checkpoint stores original category JSON, job/queue settings, queue schema/index definitions, and migration history. It allows rollback to restore type hashes/custom serializer output and schedules removed for readiness-driven processing without guessing. - Fresh v3 installations have no v2 checkpoint. An interrupted upgrade reuses its original checkpoint; a later upgrade after rollback archives the used checkpoint and captures a fresh baseline. Existing v3 databases without a checkpoint cannot recover their original settings through automatic capture. If a checkpoint is missing, establish the original settings and prepare an installation-specific rollback rather than inventing checkpoint contents. - Before an actual rollback, stop all producers/workers, resolve active leases with v3, and review broker topology/backlog recovery and persisted payload compatibility. Prepare the v2 code/configuration as well as the SQL. - The rollback transaction reverses the six job-table renames, lease columns/indexes, activation/processing-settings storage, category key/value, startup markers, and the listed v3 migrations. It restores captured configuration and maps readiness-run initiators to v2's `Schedule`, while preserving current messages, run history, extra runs, next-run parameters, and runtime execution details. It archives incompatible health-check results/statistics and v3-only control events in the checkpoint before removing them from active tables; v2 refreshes the cleared health-check caches. - Do not bypass a refusal for unknown migrations, changed queue/filter schemas/indexes or identities, conflicting tables, or unresolved leases. New v3 processors/named queues, application migrations, and broker changes need their own reversal steps. The rollback scripts preserve current messages and history; they do not recreate deleted data or undo external side effects. - Test the actual v2 application startup and representative work after rollback. Report whether checkpoint capture, SQL execution, broker recovery, and v2 restart were prepared, tested, or actually completed. ## 14. Validate behavior, not only compilation Build every affected host/library and update relevant tests according to repository instructions. A docs-only migration plan is not a validated application upgrade. Use isolated stores/brokers for checks that mutate data. Select applicable checks from this list: - The latest v2 has actually been deployed and has completed database/queue initialization in every target environment before the v3 rollout. - If rollback is part of the requested work, validate v2 settings/category restoration, v2 application startup, persisted payloads, and broker recovery using the automatically captured PostgreSQL/SQL Server checkpoint. - DI validation catches no root-provider or singleton captures of scoped Nexus services. Custom replacements can resolve their scoped dependencies. - Existing job/queue names, disabled/pause states, parameters, history, and extra runs survive migration. Saved schedules are preserved except for the explicitly agreed changes; every accepted frequent-polling conversion has both its code default removed and its saved schedule cleared in the target environment. - A readiness-driven job starts after committed enqueueing and after a delayed/retrying message becomes eligible. A scheduled-only job waits for its schedule unless explicitly started. - Frequent immediate job-start callers retain the fast requester path without added per-call extra runs. For wake-up signals, validate processing of persisted work and recovery from missed/coalesced signals. Starts specifically requiring persistence use durable extra runs and survive worker downtime; hosts using `RunAllJobsPendingSinceLastCallAsync()` do not rely on events. Event-only producers reach the correct remote job name. - With EF, entity-plus-database-message writes commit together; rollback and unsaved-context disposal expose neither message nor readiness. Failed saves can retry. Test generic/non-generic/batch paths actually used by the application. - Standalone publishing persists before returning. External/API delivery is not assumed to roll back with EF. - Representative persisted polymorphic data deserializes with the intended hash policy and unchanged salt. - Re-enqueue during a lease produces `AwaitingRelease`; a second worker cannot lease it until release; the latest payload/date/priority/status wins. Check previous-message behavior if the application relies on it. - Broker bridges report healthy and readiness acknowledgement never consumes the processing copy. Direct producers use current topology. Old Redis/Service Bus backlogs have been drained or transferred deliberately. - Custom queue batch records, positional reads, filters, retries, and optimistic versions preserve their meaning. - API clients handle the jobs wrapper, missing counts, correlation IDs, 64-bit extra-run IDs, nullable schedules, and new status. UI/API paths and authentication work in their real hosting arrangement. - A first startup upgrades representative queues, a second unchanged startup succeeds, and adding/changing a filterable property backfills correctly without losing readable payloads. Interrupted initialization can resume if custom migrations are involved. - Optional distributed processors obey intended total concurrency, pause/resume, and retry behavior; optional named queues route explicitly and keep independent identity/history. Saved processing overrides are visible and intentional. Finish with a concise report of changed projects, behavior choices, checks performed, checks requiring an environment, temporary compatibility settings, and the concrete deployment/rollback steps. Include a per-job list of accepted schedule removals, retained schedules and their reasons, and unresolved choices. Distinguish prepared metadata changes from ones actually applied. State the target package versions and how any application-specific contracts were verified. Do not report a broker migration or live deployment as completed when only its code/configuration was prepared. ## Source references When a custom integration exposes an edge case, inspect the exact target version rather than extrapolating from the rename table. Use matching source if accessible, otherwise inspect/decompile the installed assemblies as described above. Useful entry points: | Question | Assembly and types to inspect | | -------- | ----------------------------- | | Queue-job defaults, saved schedules, durable starts | `CommerceMind.Nexus.Abstractions.dll`: `IQueueJobBase`, `IJobMetaDataRepository`, `JobMetaData`; `CommerceMind.Nexus.dll`: `JobMetaDataRepositoryBase`, `JobExecutorService`, and `JobStartRequester`. Check callers and default interface member implementations too. | | Registration lifetimes and generated jobs | `CommerceMind.Nexus.dll`: `NexusServiceCollectionRegistrationExtensions`; `CommerceMind.Nexus.Abstractions.dll`: `JobDescriptor` and `GeneratedQueueJob`; registration extensions in the selected provider package. | | Activation and recovery | `CommerceMind.Nexus.dll`: `QueueActivationReconciler`, `JobExecutorService`, `DatabaseQueueActivationStoreBase`; the selected provider's activation-store implementation. | | EF transaction interception versus immediate operations | `CommerceMind.Nexus.EntityFramework.dll`: `EntityFrameworkTransactionInterceptor`; `CommerceMind.Nexus.dll`: `ImmediateQueueOperationCommitter`; `CommerceMind.Nexus.Abstractions.dll`: `ImmediateQueueOperationScope`. Follow operation commit and after-commit notification paths. | | Custom queue implementations, batches, and options | `CommerceMind.Nexus.Abstractions.dll`: `IEnqueuer`, `QueueItemUpdateOperations`, and `QueueSystemOptions`, plus the actual implementation in the storage/broker package. | The following links are optional. GitHub links point to `main`, which may differ from the installed package; use a matching release revision when available: - [Job contracts](https://github.com/Commerce-Mind-Nexus/Nexus/blob/main/src/job-engine/Nexus.Abstractions/Jobs/IJobMetaDataRepository.cs), [queue-job contracts](https://github.com/Commerce-Mind-Nexus/Nexus/blob/main/src/job-engine/Nexus.Abstractions/Jobs/Queueing/IQueueJobBase.cs), and [job registration](https://github.com/Commerce-Mind-Nexus/Nexus/blob/main/src/job-engine/Nexus/Jobs/Extensions/NexusServiceCollectionRegistrationExtensions.cs). - [Queue contracts](https://github.com/Commerce-Mind-Nexus/Nexus/blob/main/src/job-engine/Nexus.Abstractions/Queueing/IEnqueuer.cs), [batch updates](https://github.com/Commerce-Mind-Nexus/Nexus/blob/main/src/job-engine/Nexus.Abstractions/Queueing/QueueItemUpdateOperations.cs), and [queue options](https://github.com/Commerce-Mind-Nexus/Nexus/blob/main/src/job-engine/Nexus.Abstractions/Queueing/QueueSystemOptions.cs). - [EF integration](../queues/entity-framework.md), [queue maintenance](../queues/maintenance.md), [named queues](../queues/named.md), and [distributed processors](../queues/processors.md). - [External pending storage](../queues/external-pending.md), [Admin UI](../admin-ui/index.md), and [job schedules/timezones](../jobs/index.md#schedule-timezone). --- # Hangfire vs Nexus [Hangfire](https://www.hangfire.io/) is a great library for background processing and there's a lot of similarities between Hangfire and Nexus. Nexus [Functions](../nexus-functions/index.md) was built with Hangfire as an inspiration and it's especially the Functions part of Nexus and Hangfire that have a lot of similarities. Both Nexus Functions and Hangfire support persistence and scaling out to more than one processing server and are both very easy to setup to run some code in the background. Both have a built-in UI to visualize the background tasks, both have automatic retries, etc. Nexus has a slight edge over Hangfire in that Hangfire was created before Nexus and supports .NET Framework. Whereas Nexus v3 targets .NET 10+ and follows the modern .NET patterns for configuration and initialization which makes it slightly easier to get started with. ### Differences What sets Hangfire and Nexus apart is that Hangfire uses continuations and batches for more advanced scenarios. For such cases Nexus offers [background jobs](../jobs/index.md) and [queues](../queues/index.md) (eg [virtual queues](../queues/virtual.md)) instead to orchestrate complex data flows. A batch in Hangfire corresponds to a queue in Nexus with a [batch processing job](../queues/jobs.md#processing-messages-in-batches) or [distributed batch processor](../queues/processors.md#batch-processing). And a continuation in Hangfire corresponds to multiple queues in Nexus where a processing job for one enqueues to another as an effect of processing a message. In Nexus a queue message is a core concept when you design your Nexus application. You design your data flows and background tasks around queue messages and the data they contain. In Hangfire a queue message is instead simply the arguments passed to your background task method. While that's technically quite similar since they're both serialized and stored in a database the difference has some effects on your application. With Hangfire you start with a service method that you want to call in the background. With Nexus you instead often start with what data a queue message contains and add processing of that data after the message has been created. This means that Hangfire might be a better fit when you just want asynchronous and distributed background processing internally in your application and you don't really care about the data as its own entity, you just care about passing arguments to methods. And Nexus becomes a better fit if you want other applications to enqueue messages to your queue since all queues automatically get [API endpoints for that](../queues/enqueueing.md#enqueueing-over-http-using-the-api). Differentiating between the data and the processing of that data can also have benefits in larger application where those are different concerns that you don't want to mix. In Hangfire the piece of code that wants something to happen in the background is tightly coupled with the piece of code that will actually be called in the background. In Nexus the piece of code that enqueues a message for background processing doesn't even have to be in the same application as the piece of code that does the processing of that data. --- # Hosting The service is built to allow both running as a continous background service as well as a service that periodically starts up, executes outstanding jobs and then exits. The default is to run as a continous background service but you can configure it to not use the background service: ```cs builder.Services .AddNexus() .AddFunctions(options => { options.UseBackgroundService = false; }) .AddJobs(options => { options.UseBackgroundService = false; }); ``` When setting `UseBackgroundService` to `false` you need to call `IJobExecutorService.RunAllJobsPendingSinceLastCallAsync()` and `INexusFunctionExecutorService.RunAllFunctionsPendingSinceLastCallAsync()` yourself. It will check which jobs have pending runs since the last time the method was called, run them and then return. This allows the system to run in, for example, an Azure Function or Azure Container Instance that starts periodically. :::note[Running in production] Note that it's recommended to run the background service for the production environment since it offers more granular scheduling, health checks and the admin UI + API. ::: The executor services are scoped. Resolve them inside an invocation scope and keep it alive until the returned work finishes. A singleton entry point should use `IServiceScopeFactory`; see the [FaaS example](../faas.md) and [dependency injection](dependency-injection.md). Disabling the job/function loops does not disable queue maintenance, external broker consumers, or distributed processors registered separately. Only register `AddQueueProcessors()` in continuously running processing hosts. Immediate parameterless `IJobStartRequester` events are handled only by continuous job loops; use durable extra runs for periodic hosts. ## Application type The typical use case is to host the service inside an ASP.NET Core application, but any .NET 10+ compatible application can run the scheduler and queue system. The API requires ASP.NET Core. ## InstanceId In a multi-server/instance environment Nexus keeps track of which server is doing what using `INexusInstance.InstanceId`. By default that id is the value of `Dns.GetHostName()` except if you're running in Azure App Service where a single VM can be running multiple instances of Nexus using deployment slots. In this case Nexus will calculate an instance id using the current process id and the environment variable `WEBSITE_INSTANCE_ID` that's set by the app service. If you for some reason need to run multiple Nexus applications on the same VM you need to implement your own version of `INexusInstance` where `InstanceId` is guaranteed to be unique at any given time. There should never be two applications running at the same time with the same instance id. As a last resort you can set it to a random value at startup but that decreases Nexus ability to track and heal. A stable ID lets startup reset jobs/functions left running on that instance. A changing ID prevents that immediate startup reset, so recovery depends on heartbeat-based healing or manual investigation. Never share an ID between concurrently running processes. ## Graceful shutdown The importance of graceful shutdown isn't specific to Nexus but you should give your application enough time to shutdown gracefully. When you have a Web API that's typically not something you have to think about because a shutdown only needs to finish the current HTTP requests and then it can shutdown without issues. But in the case of background jobs and processing a deploy will trigger a shutdown but you might have an important job running that shouldn't be killed in the middle of execution. All jobs are given a `CancellationToken` which will fire if the application wants to shut down, so it's important for all your jobs to respect that and exit gracefully. Read more about [that here](../jobs/index.md#cancellationtoken). Configure the host shutdown timeout to give running operations enough time to stop. For example: ```cs builder.Services.Configure(options => { options.ShutdownTimeout = TimeSpan.FromSeconds(60); }); ``` This is the time .NET waits after signaling to all background services that a shutdown has initiated until the process terminates. If the process terminates in the middle of running a job you won't know what the status was for that run. Was it done? Can it be restarted? Nexus uses startup resets and heartbeat-based healing to recover stale running state, but that recovery is not an immediate acknowledgement that application work completed. Inspect the interrupted work before manually retrying it if repetition matters. Choose a timeout based on the time your handlers need to observe cancellation and release resources. It might also be that your hosting provider doesn't respect this timeout and has a shorter timeout. Verifying the timeout you have is quite easy. You can create a background service like this: ```cs public class VerifyGracefulShutdownBackgroundService : BackgroundService { private readonly ILogger _logger; public VerifyGracefulShutdownBackgroundService(ILogger logger) { _logger = logger; } protected override async Task ExecuteAsync(CancellationToken stoppingToken) { while (!stoppingToken.IsCancellationRequested) { await Task.Delay(500); } _logger.LogInformation("Shutdown initiated"); while (true) { _logger.LogInformation("Still alive..."); await Task.Delay(500); } } } ``` And then in `Program.cs` you do: ```cs serviceCollection.AddHostedService(); ``` Deploy the new code, and then deploy the same code again. After that you check the logs for the timestamps of the `Shutdown initiated` entry and the last `Still alive...` entry. If the period between those are long enough you're good to go. What should be considered long enough is up to you, but 30 seconds is a good starting point. ### Graceful Nexus shutdown For cases when you want Nexus to shutdown before the application starts to shutdown you can use the `IGracefulShutdownService` to tell Nexus to stop all background processes. Calling `IGracefulShutdownService.InitiateShutdown()` will stop all Nexus background processes and Nexus will attempt to cancel all running jobs. You can also initiate a graceful shutdown through the API using `POST api/graceful-shutdown/initiate` but only if you've set `AllowGracefulShutdownThroughApi` to `true` when calling `AddNexus().AddApi()`. Note that in a multi-instance environment you need to call this explicitly for every instance. In a multi-server environment you might want to broadcast an event that all Nexus instances listens to and calls `IGracefulShutdownService.InitiateShutdown()`. Once you've initiated graceful shutdown there's no going back. You need to restart the application for the background processes to start again. ### Soft graceful shutdown In some cases you might want to let any jobs that has started finish before you deploy but you still want to prevent new jobs from starting. In that case you can use the soft graceful shutdown in Nexus which does the same thing as a graceful shutdown except that it won't attempt to cancel running jobs or functions. Just like graceful shutdown this is an unrecoverable state, you need to restart the application for jobs to start again. You initiate a soft graceful shutdown either through the API with `POST api/graceful-shutdown/initiate-soft` or through code with `IGracefulShutdownService.InitiateSoftShutdown()`. ## Deploying multiple versions The overlap described below assumes compatible v3 schemas and message contracts. A v2-to-v3 upgrade must stop v2 before initializing v3 because the job-table rename is incompatible. Other incompatible schema/filter changes also require coordinated deployment. See [database initialization](database.md#database-initialization). Distributed processors intentionally run workers on every instance. During an overlapping deployment, both versions can process different messages and total worker capacity temporarily increases. In-memory queues require one owning instance and should not overlap during a deployment. # Azure Web Apps [Azure Web Apps](https://azure.microsoft.com/en-us/products/app-service/web/) is a popular and easy way of deploying and managing a .NET application which works great together with Nexus. However since Azure Web Apps are built for a web/API workload and not a worker workload there's some considerations when using it with Nexus. Something to be aware of is that a deployment will start your new application version on the same machines as the current version is running on and it will start the new version before the old version is signaled to initiate shutdown. For a Web API this doesn't really matter as the new application version won't receive traffic until the deploy swaps the active application and then all traffic goes to the new version. But background services will be running on both at the same time. This isn't a problem in general but something you need to be aware of. Nexus permits only one active run of each job, but distributed processors intentionally run across instances. Different application versions can therefore process different messages concurrently. During the time of a deploy there can be two different versions of the same [job](../jobs/index.md) or [function](../nexus-functions/index.md) running. Nexus will ensure that a single job never runs at the same time in different instances, but if you've scheduled a job to run every second then the old and new version of the job will compete to start running the job during the deployment. Let's say that you have a Nexus function that looks like this: ```cs public class ExampleService { public void DoSomethingInteresting() { _logger.LogInformation("This is interesting"); } } ``` And you have calls to that service frequently scheduled using `await _nexusFunction.RunInBackgroundAsync(x => x.DoSomethingInteresting());`. Then you change the log to `"This is very interesting!"`. During the deploy of that new implementation your log can look like this: ``` [20XX-XX-XX XX:XX:01] This is interesting [20XX-XX-XX XX:XX:02] This is interesting [20XX-XX-XX XX:XX:03] This is very interesting! [20XX-XX-XX XX:XX:04] This is interesting [20XX-XX-XX XX:XX:05] This is very interesting! [20XX-XX-XX XX:XX:06] This is interesting [20XX-XX-XX XX:XX:07] This is very interesting! [20XX-XX-XX XX:XX:08] This is very interesting! ``` It's up to you to consider if this timing is an issue for you, or if it's fine that different versions of the function, job or enqueuing will run at the same time. :::note[Azure Container Apps or Google Cloud Run] Since Azure App Service/Web Apps isn't optimized for a worker/background service workload you should consider using Azure Container Apps or Google Cloud Run instead which both has great support for both worker and web API workloads. ::: --- # Instance events Nexus is built to work on multiple instances/servers at the same time in a scaled out environment. You can have as many servers running jobs as you like and the system still guarantees that a single job will only run on a single instance at any given time while distributed queue processors can process different leased messages on every instance. Leases prevent simultaneous selection during the lease period; expired leases and retries can cause repeated processing. In some cases the system needs to communicate with all running instances which is called instance events. At the heart of it is [`IInstanceEvent`](https://github.com/Commerce-Mind-Nexus/Nexus/blob/main/src/job-engine/Nexus.Abstractions/Core/InstanceEvents/IInstanceEvent.cs) and [`IInstanceEventBroker`](https://github.com/Commerce-Mind-Nexus/Nexus/blob/main/src/job-engine/Nexus.Abstractions/Core/InstanceEvents/IInstanceEventBroker.cs). `IInstanceEvent` represents an event that should be sent to either all or a specific instance. Specific implementations exist for cases such as broadcasting a request to cancel a job or notifying job hosts that queue messages may be ready. Queue-readiness events are a low-latency nudge, not the durable record of work. Nexus first writes a coalesced activation through `IQueueActivationStore`, then broadcasts the event. If delivery fails, the activation remains pending and the job engine finds it during reconciliation. This means instance events can improve enqueue-to-start latency without becoming a single point of correctness. [Activation storage](../queues/activation.md#activation-storage) is configured separately from the broker; process-local memory activation is not durable across restarts. `IInstanceEventBroker` is the interface to use when you want to listen to or broadcast these events and it's up to the implementation of it to make sure that the events are sent to the correct instance(s). ## Single instance mode The single-server broker is suitable when only one instance is running. It uses a memory based solution to register listeners and broadcast events. You register it by calling this in your DI setup code: ```cs builder.Services.AddNexus().AddSingleServerInstanceEvents(); ``` ## Multi instance mode If you have multiple instances running like in an auto scaled environment where there might be multiple servers jobs running you can't use the single instance mode. Use database polling, Azure Service Bus, RabbitMQ, Redis, Kafka, or a custom broker. Both the database polling and Azure Service Bus implementation takes an option func that lets you specify `BrokerType` which is either `InstanceEventBrokerType.ProduceAndConsumeEvents` or `InstanceEventBrokerType.ProduceEvents`. For an application that only adds messages to Nexus queues using `CommerceMind.Nexus` without registering processing you can set the broker type to `InstanceEventBrokerType.ProduceEvents`. This means that Nexus will skip setting upp polling/subscriptions for that application. ```cs builder.Services.AddNexus().AddDatabasePollingInstanceEvents(options => { options.BrokerType = InstanceEventBrokerType.ProduceEvents; }); // Or: builder.Services.AddNexus().AddAzureServiceBusInstanceEvents(options => { options.BrokerType = InstanceEventBrokerType.ProduceEvents; }); ``` :::note[CommerceMind.Nexus.ApiClient] If you're using the `CommerceMind.Nexus.ApiClient` package that [enqueues messages over the HTTP API](../queues/enqueueing.md#enqueueing-using-the-apiclient-package) you don't need to configure instance events at all for that application. ::: ### Database polling If you want to limit external dependencies you can use the database to store instance events and have your servers poll the database for new events. You configure database polling events like this: ```cs builder.Services.AddNexus().AddDatabasePollingInstanceEvents(options => { options.PollDelay = TimeSpan.FromSeconds(1); // Default polling is every second }); ``` This implementation will check if there's multiple running instances and then save a copy of the event for each running instance except for itself where it broadcasts the event directly. It knows which instances are running by using the [`INexusInstanceHeartbeatService`](https://github.com/Commerce-Mind-Nexus/Nexus/blob/main/src/job-engine/Nexus.Abstractions/Core/Heartbeats/INexusInstanceHeartbeatService.cs). Read more about [heartbeats here](./monitoring.md#heartbeats). ### RabbitMQ If you're using RabbitMQ you can use that as your instance event broken. Install the package `CommerceMind.Nexus.RabbitMQ` and configure the instance event broker: ```cs builder.Services .AddNexus() .AddRabbitMq(options => { options.UserName = "..."; options.Password = "..."; options.HostNames = ["localhost"]; // Or: options.ConnectionString = new Uri("amqp://..."); }) .AddRabbitMqInstanceEvents(options => { // This is the default value, you only need to change this if you're using // multiple Nexus applications with the same RabbitMQ instance. options.ExchangeName = "nexus_instance_events"; }); ``` ### Azure Service Bus If you're using Azure there's a NuGet package called `CommerceMind.Nexus.Azure` which uses a service bus topic to broadcast instance events. After you've installed it you configure it like this in your `Program.cs`: ```cs builder.Services.AddNexus().AddAzureServiceBusInstanceEvents(); ``` The default name of the topic is `nexus-instance-events` but that can be configured by passing an options func like this: ```cs builder.Services.AddNexus().AddAzureServiceBusInstanceEvents(options => { options.TopicName = "my-topic-name"; }); ``` By default Nexus will look for the connection string using `IConfiguration.GetConnectionString("azureServiceBus")` but you can add a loader func on the options object if you need to fetch it from somewhere else: ```cs builder.Services.AddNexus().AddAzureServiceBusInstanceEvents(options => { options.ConnectionStringLoader = (serviceProvider) => "Endpoint=sb://xxx.servicebus.windows.net/;..."; }); ``` Nexus will automatically create topic subscriptions for each instance which works best in an auto scaled environment where you don't know how many instances you will have. In order to be able to create subscriptions on-the-fly you also need to configure an administration connection string. You do that either by setting the connection string `azureServiceBusAdmin` with `IConfiguration` or set the `AdminConnectionStringLoader` property on the options object. If you're not using an auto scaled environment and you instead want to create the subscriptions yourself you can skip configuring the admin connection string and instead add `SubscriptionNameLoader`: ```cs builder.Services.AddNexus().AddAzureServiceBusInstanceEvents(options => { options.SubscriptionNameLoader = (serviceProvider) => "my-subscription"; }); ``` :::note[One subscription per server] Make sure that all instances use a different subscription when you're manually creating them. This setup only works if you have a number of dedicated machines such as VMs that are always running. ::: #### Batching events By default instance events are batched so that events that happen within 0.5 seconds are sent as a single service bus message. This is done to limit the number of messages since Azure Service Bus has a cost per message. If a 0.5 second delay doesn't work for you it's possible to change the delay like this: ```cs builder.Services.AddNexus().AddAzureServiceBusInstanceEvents(options => { options.DebounceDelay = TimeSpan.FromSeconds(5); options.MaxMessageBatchSize = 10; // The default is 10 }); ``` The `MaxMessageBatchSize` is to guard against getting a message size that is bigger than what's allowed. So if there's more than 10 events within 0.5 seconds they will get sent as mutiple messages. If you're using the standard tier service bus the limit is 256 kB but if you're using the premium tier the limit is 100 MB. ### Redis If you're using Redis you can use that as your instance event broker. Install the package `CommerceMind.Nexus.Redis` and configure the instance event broker: ```cs builder.Services .AddNexus() .AddRedisInstanceEvents(options => { options.ConnectionString = "localhost:6379"; }); ``` If you already have an `IConnectionMultiplexer` registered in DI you can reuse it instead of providing a connection string: ```cs builder.Services .AddNexus() .AddRedisInstanceEvents(options => { options.ConnectionMultiplexerFactory = sp => sp.GetRequiredService(); }); ``` The default pub/sub channel name is `nexus_instance_events` but that can be changed if you're using the same Redis instance for multiple Nexus applications: ```cs builder.Services .AddNexus() .AddRedisInstanceEvents(options => { options.ConnectionString = "localhost:6379"; options.ChannelName = "my-app-instance-events"; }); ``` Redis uses pub/sub for broadcasting, so all subscribers on the channel receive every message — no per-instance group or subscription configuration is needed. ### Kafka If you're using Kafka you can use that as your instance event broker. Install the package `CommerceMind.Nexus.Kafka` and configure both the Kafka connection and the instance event broker: ```cs builder.Services .AddNexus() .AddKafka(options => { options.ConsumerConfig.BootstrapServers = "localhost:9092"; options.ConsumerConfig.GroupId = "my-nexus-app"; options.ProducerConfig.BootstrapServers = "localhost:9092"; }) .AddKafkaInstanceEvents(); ``` The default name of the topic is `nexus_instance_events` but that can be configured: ```cs builder.Services .AddNexus() .AddKafka(options => { ... }) .AddKafkaInstanceEvents(options => { // Change this if you're using the same Kafka instance for multiple Nexus applications. options.TopicName = "my-topic-name"; }); ``` If you need separate producer/consumer configuration specifically for instance events you can override them via the options: ```cs builder.Services .AddNexus() .AddKafka(options => { ... }) .AddKafkaInstanceEvents(options => { options.ProducerConfig = new ProducerConfig { BootstrapServers = "localhost:9092" }; options.ConsumerConfig = new ConsumerConfig { BootstrapServers = "localhost:9092", GroupId = "my-nexus-app" }; }); ``` Because Kafka delivers messages to only one consumer per consumer group, every instance must use a **unique** `GroupId` in order to receive all broadcast events. A common pattern is to include the hostname or a random ID in the group ID: ```cs builder.Services .AddNexus() .AddKafka(options => { options.ConsumerConfig.BootstrapServers = "localhost:9092"; options.ConsumerConfig.GroupId = $"nexus-instance-events-{Environment.MachineName}"; options.ProducerConfig.BootstrapServers = "localhost:9092"; }) .AddKafkaInstanceEvents(); ``` ## Custom application instance events You can use the Nexus instance event infrastructure to broadcast events within your application. To do this, define your own event class that implements the `IApplicationInstanceEvent` interface and define handlers that implement the `IApplicationInstanceEventHandler` interface. You can then use the `CommerceMind.Nexus.Abstractions.Core.InstanceEvents.IApplicationInstanceEventPublisher` to publish events that will be broadcast to all running instances. Eg: ```cs public class MyEvent : IApplicationInstanceEvent { public string? MyProperty { get; set; } } public class MyEventHandler : IApplicationInstanceEventHandler { public async Task HandleAsync(MyEvent myEvent) { // Do something interesting with the event } } builder.Services.AddScoped, MyEventHandler>(); var app = builder.Build(); await using var scope = app.Services.CreateAsyncScope(); var publisher = scope.ServiceProvider.GetRequiredService(); await publisher.PublishAsync(new MyEvent()); ``` Application event handlers run in a dependency injection scope and can use scoped services. These events use the configured broker's delivery semantics; they are notifications, not a durable application work queue. Use a Nexus queue when the work must be persisted and retried. ## Subscribing to instance events Use `IInstanceEventBroker.Subscribe(Func)` for infrastructure subscriptions and dispose the returned `IInstanceEventSubscription` when the subscriber is disposed. Handlers are asynchronous. A long-lived subscriber should create an operation scope when it needs scoped Nexus/application services instead of capturing them in the broker. ## Writing your own broker If you don't feel like any of the above options works for you then you can always create your own implementation of [`IInstanceEventBroker`](https://github.com/Commerce-Mind-Nexus/Nexus/blob/main/src/job-engine/Nexus.Abstractions/Core/InstanceEvents/IInstanceEventBroker.cs). An important thing to think about is that `IInstanceEvent` has an `InstanceId` property that can be `null` or have a value. If it's `null` then the event should be sent to all instances. But if it's not null it should only be published as an event on the instance where [`INexusInstance.InstanceId`](https://github.com/Commerce-Mind-Nexus/Nexus/blob/main/src/job-engine/Nexus.Abstractions/Core/NexusInstance/INexusInstance.cs) is the same as `IInstanceEvent.InstanceId`. A custom broker can receive a broadcast on every instance and filter by `InstanceId` before dispatching it. Implement `Subscribe()` and await asynchronous handlers. The `InstanceEventSubscriptions` helper manages handler registration, dispatch, and disposable subscriptions: ```cs private readonly InstanceEventSubscriptions _subscriptions = new(); public IInstanceEventSubscription Subscribe(Func handler) => _subscriptions.Subscribe(handler); // In the transport receive path, after filtering the target instance: await _subscriptions.PublishAsync(instanceEvent); ``` The helper handles local subscribers; the broker still owns serialization, transport, instance targeting, and shutdown. Do not settle a transport notification before required asynchronous handling has completed. When you're done with the implementation all you need to do is register it like this: ```cs builder.Services.AddSingleton(); ``` --- # Nexus and JSON Nexus uses `System.Text.Json` by default to serialize persisted messages, job parameters, function arguments, instance events, and other state. Applications using Newtonsoft.Json converters can install `CommerceMind.Nexus.NewtonsoftJson` and call `AddNexus().AddNewtonsoftJson()` instead. Choose the serializer deliberately and check existing persisted data before switching. ## Customizing serialization Nexus serialization is controlled by its serialization interfaces and options. ASP.NET Core controls the outer HTTP API models, while Nexus controls the queue-message JSON inside them. For example, the outer model can use camelCase while the message retains PascalCase: ```json { "id": 4823771, "status": "Processed", "retryCount": 0, "message": { "Id": null, "SomeProperty": null, "ExampleEnum": "Value1" } } ``` Configure Nexus's System.Text.Json converters through: ```cs builder.Services.AddNexus().ConfigureJsonOptions(options => { options.SerializerOptions.Converters.Add(new MyCustomConverter()); }); ``` Specialized interfaces such as `IQueueMessageSerializer`, `IJobParametersSerializer`, and `INexusFunctionArgumentsSerializer` can be replaced when different formats are required. Operational serializers are scoped and may use scoped application dependencies; see [dependency injection](dependency-injection.md). ## Interfaces, abstract types, and type hashes When polymorphic data needs a concrete .NET type at deserialization time, Nexus can write `$type` metadata and a `$typehash`. This is not required on every ordinary queue message; the queue's known message contract normally supplies its type. In v3, `JsonOptions.WriteTypeHash` and `RequireTypeHash` both default to `true`. When the System.Text.Json polymorphic converter reads explicit type metadata, it requires a matching hash. Custom writers must place `$typehash` before `$type`, and both before ordinary properties. Use the Nexus serializer to produce this metadata rather than constructing it manually. Keep the same `TypeHashSalt` on every host reading the same data. Changing it invalidates existing hashes. If a salt change is necessary, use a compatible `ISerializationTypeHashCalculator` during the transition so old and new data can both be read. `ISerializationBinder` controls the mapping between .NET types and their serialized names. `ISerializationTypeHashCalculator` calculates hashes for those names. Preserve these mappings when renaming application types used in persisted data. ## Older JSON without hashes Default v2 System.Text.Json writers already included hashes. Older or customized JSON may have `$type` without `$typehash`, particularly data from v1 or Newtonsoft-compatible writers. Process or migrate that data before enabling the v3 requirement, or temporarily configure: ```cs builder.Services.AddNexus().ConfigureJsonOptions(options => { options.RequireTypeHash = false; // Leave WriteTypeHash enabled so new System.Text.Json data has hashes. }); ``` This relaxes validation for the STJ polymorphic converter; it is not a fix for unreadable renamed types or a serializer switch. Enable `RequireTypeHash` again after the old data has been removed or migrated. Do not disable hash writing while requiring it on reads, as that can make newly written polymorphic data unreadable. --- # Metrics with Open Telemetry Nexus exposes a number of metrics using Open Telemetry. Open Telemetry is an open standard for exposing metrics and telemetry that can be plugged into almost any monitoring service such as Azure Application Insights, Splunk, Elastic APM etc. If you're using Azure Application Insights/Azure Monitor for example you can install the Open Telemetry exporter for Azure Monitor: [https://learn.microsoft.com/en-us/azure/azure-monitor/app/opentelemetry-configuration?tabs=aspnetcore](https://learn.microsoft.com/en-us/azure/azure-monitor/app/opentelemetry-configuration?tabs=aspnetcore) Install the NuGet package `Azure.Monitor.OpenTelemetry.AspNetCore` and then call: ```cs builder.Services .AddOpenTelemetry() .UseAzureMonitor(options => { options.ConnectionString = ""; }); ``` This will export the metrics and telemetry data from Nexus into Azure Monitor. You'll be able to see counters such as how many queue messages are processed and how many fail. Each job run is exported as a trace with a start and end in the same way as Application Insights represents an HTTP request. If you also use the Application Insights SDK you'll see all the database queries or HTTP requests that happens inside that job run as well. This is not specific to Application Insights though as Nexus has no knowledge of that. It only exposes the telemetry in the Open Telemetry standard format and then its up to you to send that data to whichever APM you're using. Queue job and distributed processor message counters include a `queue.name` tag. Use it to distinguish independently processed named queues sharing the same implementation class. --- # Monitoring Nexus provides health checks, instance heartbeats, job-run logs/progress, queue statistics, and OpenTelemetry metrics. Use [health checks](../healthchecks/index.md) for the current operational status and [metrics](metrics.md) for throughput and timing over time. ## Health checks Jobs, queues, and functions register health checks with Nexus. When the API's `RegisterHealthChecksAsAspNetCoreHealthChecks` option is enabled (the default), an ASP.NET Core host can expose them through: ```cs app.MapHealthChecks("/health"); ``` Job failures, long-running jobs, failed functions, and queue errors are visible in the Admin UI. Job and queue attributes/metadata can customize the health status used for failures. Historical transitions are available through [health-check statistics](../healthchecks/statistics.md). For custom checks, non-web hosts, and publishers, see [health-check configuration](../healthchecks/index.md). Publishers needing scoped Nexus dependencies should use `AddScopedHealthCheckPublisher()`; see [dependency injection](dependency-injection.md#custom-registrations-and-health-publishers). [Teams](../healthchecks/teams.md) and [Slack](../healthchecks/slack.md) integrations publish failures and recovery notifications. ## Heartbeats Nexus reports instance heartbeats so other hosts can distinguish active instances from ones that have stopped reporting. `INexusInstanceHeartbeatService` in `CommerceMind.Nexus.Abstractions.Core.Heartbeats` exposes `GetNexusInstanceStatusesAsync()` for inspection. Keep `INexusInstance.InstanceId` stable and unique among concurrently running instances. Heartbeats and startup resets help recover jobs/functions left running after a crash. They do not undo application side effects; inspect an interrupted operation before retrying it when repetition could matter. See [hosting](hosting.md#instanceid). ## Queue readiness and distributed workers A queue can contain pending messages while processing is disabled, paused on error, waiting for activation delay, or waiting for a cron schedule. Queue activation state is separate from message count and job-run state. See [queue activation](../queues/activation.md). External pending providers register `NexusExternalQueueReadiness_{ProviderId}` health checks. An unhealthy bridge can prevent prompt starts even while the broker retains the messages. Verify topology, subscriptions/groups, and access permissions as described in [external pending storage](../queues/external-pending.md#readiness-bridges-and-direct-producers). Distributed processors do not create job runs. Inspect their queue's Processing tab or `/api/queues/{queueName}/processor` for active workers, in-flight messages, instance reports, and recent errors. These controls and [processing settings](../queues/processing-settings.md) belong to the new Admin UI. Message counters include `queue.name` to distinguish queues sharing an implementation. Queue statistics describe status transitions, not current backlog. The API omits queue counts unless requested; missing counts do not mean zero messages. Combine counts, error status, leases, and worker activity when diagnosing processing delays. --- # 3.0.1 _(3.0.0 didn't contain a rename of QueueAttribute.KeepProcessedMessagesFor, upgrade directly to 3.0.1 and skip 3.0.0)_ _2026-11-01_ Nexus 3 brings in a lot of great improvements. It's a performance focused released with a lot of good stuff in it. Feel free to reach out to us at Commerce Mind to discuss how these new features can help you and how we recommend using them. Nexus 3 contains breaking changes. Read the [v2 to v3 upgrade guide](upgrade-guide.md#from-v2-to-v3) before upgrading. ## Performance and database initialization - Database startup now remembers completed migrations and queue schema definitions. Unchanged starts skip migration-history and queue schema checks; new queues and changed filterable properties are initialized independently. Manually changed Nexus schemas are no longer repaired automatically. Should be a drastic improvement in startup speed for most. - Improved throughput for in-memory queues, multithreaded queue jobs, and external pending storage, including Azure Service Bus enqueueing and batched completion/idempotency updates. - Queue maintenance now runs independently from the job loop, wakes for known retry and lease deadlines, and deletes retained data in bounded batches. See [queue maintenance](../queues/maintenance.md) for tuning options. - Queue lease indexes now match priority leasing order. Existing SQL Server, PostgreSQL, and SQLite indexes are migrated automatically; allow time and database space for large queues. Example lease speed up: from 1,141ms to 13ms, thanks again [Patrick](https://github.com/patfog)! - Job tables are automatically renamed from `ScheduledJob` to `Job` names. Stop v2 instances before starting the v3 migration. - Nexus can now process thousands of messages per second with the combination of [external pending storage](../queues/external-pending.md) and [distributed queue processors](../queues/processors.md) ## Jobs and queue processing - Queue jobs no longer require a schedule. Without a schedule they start when messages become ready; with a cron schedule they only start on that schedule. - `EnqueueAsync()` now requests coalesced queue activation after messages are committed. The `EnqueueAndStartProcessingAsync()` methods have been removed. - Added database and Redis queue activation storage, and an in-memory option for single-instance applications. - Added `ActivationDelay` to collect a short burst of messages before starting processing. Unlike `ProcessingDelay`, it doesn't delay each individual message. - Added [distributed queue processors](../queues/processors.md) (`IQueueProcessor` and `IBatchQueueProcessor`) for processing messages across application instances without an explicit job. - Added [named queues](../queues/named.md), making it possible to have multiple independent queues for the same message type without creating generic subclasses. - Generated queues now use the named queue infrastructure instead of runtime-generated message and job subclasses. Existing generated-queue producers need explicit destinations or a routing entry point. - Processing threads, workers, lease time, lease size, and batch size can now be configured per queue through the API and the new Admin UI, depending on the processing implementation. - Renamed schedule-neutral `ScheduledJobXxx` APIs to `JobXxx`, including `AddScheduledJobs()` to `AddJobs()` and `[ScheduledJob]` to `[Job]`. Queue jobs now use `IQueueJob` and `IBatchQueueJob`; ordinary scheduled jobs still use `IScheduledJob`. - Added [schedule timezone configuration](../jobs/index.md#schedule-timezone). Cron schedules still use UTC by default. - Immediate parameterless starts through `IJobStartRequester` now use fast, best-effort instance events. Use `IJobMetaDataRepository.ScheduleExtraRunAsync()` when the start must be persisted. Requests can also be sent from applications without local job registration or `AddJobs()`. - Queue jobs no longer count pending messages or report progress automatically. Use `AutomaticProgressMode` to opt into a processed count or percentage progress. Updates are aggregated per lease or batch. ## Queues and integrations - Re-enqueueing a leased message in database and in-memory queues now uses `AwaitingRelease`. The current processor keeps its lease, and the latest enqueue's payload, priority, processing date, and destination status are applied when it finishes. This prevents another processor from picking up the newer version while the previous version is still being processed. - Added durable readiness bridges for Redis, RabbitMQ, Kafka, and Azure Service Bus external pending storage. RabbitMQ now uses a fan-out exchange with processing and readiness queues; Azure Service Bus uses a topic with two subscriptions. Redis keys now use a shared cluster hash tag. See the [upgrade guide](upgrade-guide.md#external-pending-storage). - Reworked the [Entity Framework integration](../queues/entity-framework.md) to use the current dependency injection scope instead of global `DbContext` tracking. Database and in-memory enqueue operations now follow the scoped unit of work through all enqueuer interfaces. - Queue and job registration now combines contributions from multiple libraries at `Build()`, including named/generated queues and their processing bindings. Repeated configuration calls share options and validate duplicate names and conflicting processors. - Virtual queue transformers can route to named queues or several destinations, including destinations using the same message type as the virtual queue. - `QueueSystemOptions.DisableHistory` now overrides queue-history settings stored in the database. - Fixed SQL Server filtering for `List` properties and bulk enqueueing with filterable `Guid` properties. - Removed the retired Teams incoming-webhook integration. Use the Teams bot or bridge integration instead. ## Admin UI and API - The new Admin UI is now the default. The legacy UI is temporarily available through `UseLegacyVersion`. - The Admin UI is registered separately from the API through `AddAdminUI()`, with an explicit API base URL. - Added live job progress to the jobs list, job details, and current-run views in the new Admin UI. - Added queue processing controls and instance/worker status for distributed processors in the new Admin UI. Both UI versions show optional queue-job schedules, pending activation, and activation delay. - Improved the queues list, display of validation errors, and local time display in the new Admin UI. - Updated the HTTP API's job list, queue counts, historical-run identity, and extra-run IDs. Custom clients need to be updated; queue counts are only loaded when requested. ## Other breaking changes - All Nexus packages now require .NET 10. - Most operational Nexus services are now scoped instead of singleton, making scoped dependencies available in jobs, functions, queues, health checks, and integrations. Singleton application services need to create a scope when calling these services. - System.Text.Json polymorphic payloads now require a valid type hash by default. This mainly affects applications with customized serialization or older persisted JSON. - Removed obsolete public APIs and v1 compatibility code. Direct upgrades from v1 are no longer supported; upgrade through the latest v2 release first. # 2.58.0 _2026-09-28_ - In the [new Admin UI](../admin-ui/index.md#admin-ui-versions) it's now possible to [upload files as parameters to jobs](../jobs/parameters.md) - Various improvements to the [new Admin UI](../admin-ui/index.md#admin-ui-versions) - Since the webhook integration with Microsoft Teams has been deprecated by Microsoft there's now a new [Teams integration](../healthchecks/teams.md) # 2.57.2 _2026-08-06_ - Performance improvement in job-start loop when setting job loop delay to less than 1s - Fix potential memory leak in in-memory queues # 2.57.1 _2026-05-20_ - Minor fixes and tweaks to the new admin UI # 2.57.0 _2026-05-19_ - Add a **new admin UI** which can be used by [setting an option](../admin-ui/index.md#admin-ui-versions) on the Admin UI options during configuration # 2.56.0 _2026-05-13_ - Support idempotency and message ids in the [Redis](../queues/redis.md) external pending storage - Add `[IdempotencyKey]` [as a way to control how idempotency is calculated](../queues/index.md#idempotent-messages) # 2.55.0 _2026-05-12_ - Make it possible for Nexus to process messages directly from RabbitMQ, Azure Service Bus, Redis, and Kafka using [external pending storage](../queues/external-pending.md) - Add Redis package for instance events and external pending storage - Improve performance of memory queues # 2.54.1 _2026-05-08_ - Fix admin UI taking over all paths that starts with the admin UI path, eg taking over `/nexus-path` when the admin UI is at `/nexus`. # 2.54.0 _2026-05-08_ - Add `VirtualHost` to RabbitMQ options. # 2.53.0 _2026-05-08_ - Add new package `Nexus.Mcp` to expose an MCP endpoint to let AI agents work with Nexus. See [docs about MCP](../api/mcp.md) # 2.52.0 _2026-05-05_ - Prevent jobs scheduled at every second from being able to run more frequent than once per second when the job loop is configured to run more often than once per second. - Add Entity Framework integration for IEnqueuer where it waits for EFs transaction to commit queue messages together with EF - Add more logging in Azure Service bus queue ingestion # 2.51.0 _2026-03-10_ - Add option `MaxStartupParallelization` which will attempt to create queue tables in parallel. Use this if faster startup time is important and you have a lot of queues. - Fix issue causing job mutexes to be released when they shouldn't. # 2.50.3 _2026-01-30_ - Don't wait for Rabbit connection before starting Nexus when using RabbitMQ # 2.50.2 _2025-12-16_ - Fix issue with property default values (eg false and 0) in partial messages # 2.50.1 _2025-12-10_ - Use incremental backoff delay when job loop fails due to database errors instead of a fixed 30s delay # 2.50.0 _2025-12-10_ - Allow [configuring the job loop delay](../jobs/index.md#job-loop-delay) - Avoid null referecnce exception in health check that can happen during high parallelism # 2.49.3 _2025-11-28_ - Remove dependency on System.Linq.Async - Show logs with level Critical as red in job log viewer - Handle health check result description being null # 2.49.2 _2025-11-20_ - Force a new async context when starting a job from an instance event to prevent enrolling the job run in any transactions - Don't compile lambda when expression is a constant expression since compiling is expensive # 2.49.1 _2025-11-18_ - Don't block thread by event handler when receiving instance event # 2.49.0 _2025-11-11_ - Make default parameters and next run parameters editable in admin UI # 2.48.6 _2025-11-05_ - Make all IHostedServices use try-catch in StopAsync() - Mark a queue job run as NothingToDo when all messages where skipped/NotProcessed # 2.48.5 _2025-10-14_ - Wait for 5m instead of 1m before warning about a job that hasn't started yet since some jobs might not start right away when many jobs are running # 2.48.4 _2025-10-03_ - Fix issue causing some historical job data not to be deleted correctly # 2.48.3 _2025-09-24_ - Don't retry SQL Server deadlock exceptions when an external transaction is set - Don't scan the same assembly twice for the same queue messages - Add more/longer retries when RabbitMQ connection fails # 2.48.2 _2025-09-09_ - Fix issue with dropping unused queue indexes in Postgres and Sqlite # 2.48.1 _2025-09-05_ - Make it possible to skip sending Degraded health checks to Teams # 2.48.0 _2025-08-22_ - Make it possible to change the health status level a job or queue has when the job is failing or when the queue has error messages in it - Add health check publisher to [Microsoft Teams](../healthchecks/teams.md) # 2.47.5 _2025-06-26_ - Fix issue causing `[Queue(AutomaticRetryAfter = "xxx")]` not to be correctly used # 2.47.4 _2025-06-24_ - Fix support for scoped services in Nexus Functions # 2.47.3 _2025-06-09_ - Add locks when acknowledging messages with RabbitMQ - Don't load the same log from Elastic multiple times when looking at logs for a current run # 2.47.2 _2025-05-04_ - Fix issue causing the Elastic job log reader not to return all logs # 2.47.1 _2025-05-04_ - Downgrade Elastic from 9 to 8 in `CommerceMind.Nexus.Elastic` # 2.47.0 _2025-05-04_ - Add [queue message validation](../queues/validation.md) # 2.46.0 _2025-04-26_ - Add `CommerceMind.Nexus.Elastic` NuGet package to [load job logs from Elasticsearch](../jobs/logs.md#elasticsearch) instead of the database - Use explicit lock in SQL Server when leasing functions for execution # 2.45.0 _2025-04-26_ - Prevent admin UI job list from flickering when scrolling - Allow updating processed message retention in the admin UI - Make it possible to add a delay on a queue before a job is allowed to process a message - Handle exceptions when service provider fails to create a job instance - Increase query timeout for operations that update all or most items in a queue - Fix url to queue webhook endpoint in admin UI - Handle errors that might occur when organizing queue messages - Handle ProcessedMessageRetention being TimeSpan.MinValue in memory queues # 2.44.4 _2025-04-07_ - Include cookies for different sub domains in requests made by the Admin UI to make auth easier to implement # 2.44.3 _2025-04-05_ - Fix issue causing some memory queue changes not to get persisted to fallback storage properly - Ignore any DbConnectionScope when explicitly asking for a new db connection # 2.44.2 _2025-04-03_ - Only register ApplicationInstanceEventBroker if instance events have been registered # 2.44.1 _2025-04-02_ - Fix issue during startup when instance events are not configured, eg when only using Nexus API enqueuers # 2.44.0 _2025-04-02_ - Make it possible to use Nexus instance events infrastructure for [application specific events](./instance-events.md#custom-application-instance-events) - Use retries when cleaning up historical job runs # 2.43.0 _2025-04-02_ - Add queue export to Excel in Admin UI - Make it possible to control how ServiceBusClient is created in CommerceMind.Nexus.Azure - Throw better error when there are multiple queues with the same name # 2.42.0 _2025-03-22_ - Make it possible for virtual queue transformers to [return an `EnqueueContext`](../queues/virtual.md#enqueuecontext) to control things like processing date and priority - Add `EnqueueContextFactory` option when enqueueing from [Azure Service Bus](../queues/azure.md), [Rabbit MQ](../queues/rabbitmq.md) and [Kafka](../queues/kafka.md) - Fix issue with job progress message disappearing # 2.41.1 _2025-03-18_ - Support JSON arrays in Azure Service Bus messages - Prevent unnecessary delay when requesting a job to start - Fix issue with job progress total disappearing # 2.41.0 _2025-03-13_ - Allow specifying `MaxConcurrentCalls` for Azure Service Bus # 2.40.1 _2025-03-13_ - Run Kafka message processing in dedicated tasks # 2.40.0 _2025-03-12_ - Handle changes in casing of job names - Allow setting new priority and update message data when returning NotProcessed() for a message - Change default Admin UI path from /admin to /nexus and redirect /admin to /nexus - Skip setting up Azure Service Bus listener and log if not properly configured instead of throwing - Don't log TaskCanceledExceptions caused by service shutdown # 2.39.0 _2025-03-10_ - Use Ignore instead of Null as key for Kafka consumer - Allow passing a message deserializer to EnqueueFrom() when declaring a RabbitMQ queue # 2.38.0 _2025-03-10_ - Generate explicit OpenApi definitions for queue webhook endpoints # 2.37.0 _2025-03-08_ - Add `ProcessResults.NotProcessed()` for situations where you can't process and message and you want to return it to the queue untouched - Ensure that job progress table can't have multiple rows for a single job - Return data from update queries and fill job meta data cache to avoid having to fetch the data again # 2.36.0 _2025-03-07_ - Make it possible to force update/delete messages in Admin UI - Fix deserializing nullable number type - Support passive queues in RabbitMQ # 2.35.4 _2025-03-07_ - Include job meta data version in instance event to correctly purge the cache # 2.35.3 _2025-03-07_ - Fix handling of multiple progress entries for the same job # 2.35.2 _2025-03-07_ - Improve caching of job meta data - Add option to turn off v1 compatbility checks - Avoid unnecessary message parsing for memory queues # 2.35.1 _2025-03-06_ - Avoid creating a job instance to check `StopProcessingOnError` if that property is not defined on the type - Show message deserialization errors in Admin UI instead of crashing # 2.35.0 _2025-03-06_ - Make it possible to dynamically [generate queues and queue jobs](../queues/generated.md) - Show filtered queue counts for jobs with message filter in Admin UI - Make PauseOnError work with filtered queue jobs - Make it possible to choose how many queue messages to see per page in Admin UI - Health checks for jobs now becomes unhealthy if it hasn't started on schedule - Fix queue counts disappearing in job list when scrolling - Refresh queue item list when navigating - Refresh queue status counts when updating queue item(s) in Admin UI - Allow queue processing retries to be scheduled in 10s or less from now - Increase query timeout when enqueueing a lot of messages # 2.34.1 _2025-02-25_ - Avoid making HTTP request in Admin UI that doesn't fetch any data - Respond with 400 Bad Request instead of 500 in queue webhook when the incoming message is malformed - Don't create unused RabbitMQ channel when using RabbitMQ for instance events # 2.34.0 _2025-02-19_ - Make it possible to have dynamic (untyped) content in queue messages [using `IDynamicQueueMessage`](../queues/index.md#dynamic-message-content) # 2.33.0 _2025-02-09_ - Include queue message type in Open Telemetry metrics - Extract UI into separate NuGet package so that you can host the UI [in a separate application](../admin-ui/index.md#hosting-the-ui-on-its-own) # 2.32.1 _2025-02-06_ - Fix Postgres returning null when any property in the message is null # 2.32.0 _2025-02-01_ - Fix alignment issues for buttons on queue page - Fix issue with hidden deleted jobs and queues in the admin UI - Make it possible to hide job and queue "More" settings with the escape button - Make it possible to start the queue job from the queue page - Fix filtered "Message id" text field on refreshing queue page # 2.31.2 _2025-01-24_ - Start queue jobs where job start has been requested before other jobs - Handle multiple manual job run requests executing at the same time # 2.31.1 _2025-01-24_ - Fix issue with EnqueueAndStartProcessingAsync() causing intermittent errors - Fix SQL Server issue with upgrading from an old version # 2.31.0 _2025-01-22_ - Add [CommerceMind.Nexus.Kafka](../queues/kafka.md) which enables instance events over Kafka as well as automatically enqueueing to Nexus queues from Kafka topics - Don't wait for the next job loop tick when enqueueing a message and requesting processing to start # 2.30.0 _2025-01-21_ - Expose more options to RabbitMQ binding object # 2.29.0 _2025-01-18_ - Fix issue causing reloads in Admin UI when scrolling long jobs or queues lists - Make it possible to update a message as part of processing it - Make it possible to hide jobs and queues in the Admin UI - Show notification that data could not be refreshed instead of showing error screen - Cache frequently executed job meta data query # 2.28.0 _2024-12-20_ - Generate XML comment files for NuGet packages - Retry failed requests in the Admin UI - Only load queue status counts for visible queues in the Admin UI - Base Admin UI polling on when the last request ended instead of started to prevent overloading the server when it responds slowly # 2.27.2 _2024-12-17_ - Fix issue with invalid existence check on SQL Server queue column # 2.27.1 _2024-12-15_ - Fix issue with Azure Postgres not always returning all columns in information_schema.columns # 2.27.0 _2024-12-15_ - Add [CommerceMind.Nexus.RabbitMQ](../queues/rabbitmq.md) and make it possible to use [RabbitMQ for instance events](./instance-events.md) - Make it possible to use [dynamic job mutexes](../jobs/index.md#job-parallelism-and-mutexes) - Make it possible to read queue messages by a list of message ids - Add search/filter to header bar in the Admin UI - Make it possible to retry messages instead of marking them as Abandoned - Make it possible to limit which instance a job should run on through code and in the Admin UI - Fix issue that caused enable/disable buttons for jobs not to be responsive in the Admin UI - Support [top level dictionaries in partial messages](../queues/index.md#partial-messages) - Allow queue jobs to control [the job result message](../queues/jobs.md#controlling-the-job-result-message) - Make it possible for a [queue job to filter which messages it wants to process](../queues/jobs.md#filtering-messages) - Make it possible to [filter on queue properties](../queues/filterable.md) # 2.26.0 _2024-12-05_ - Show dates as labels instead of time when showing more than latest 24h in statistics - Hide sorting controls on queues list until hovered - Add automatic webhook endpoint for all queues, [see docs](../queues/webhooks.md) - Wrap ActivitySource usage in order to trap exceptions during dispose, fixes issue where Elastic.APM causes jobs to be reported as failed # 2.25.2 _2024-12-02_ - Fix exception caused by queues that no longer exist in code - Correctly handle status change statistics for idempotent queues # 2.25.1 _2024-12-02_ - Retry TimeoutExceptions that occur in the database layer # 2.25.0 _2024-12-02_ - Make it possible to sort jobs and job categories in the Admin UI - Don't show processed count in queue stats if processed messages are deleted since it's confusing to see "Processed: 0" when the graph says lots of processed messages - Only show status counts below graphs of the statuses shown in the graph - Optimize performance of updating statistics graphs in Admin UI # 2.24.1 _2024-11-30_ - Generate random color for custom statuses in queue statistics graphs - Fix issue causing queue meta data to be reset when updating meta data # 2.24.0 _2024-11-30_ - Make it possible to control order and items in the Admin UI main menu. [See docs](../admin-ui/index.md). - Add statistics/graphs over queue message status changes. [See docs](../queues/statistics.md). - Allow setting which status a message should get when reaching max retries. [See docs](../queues/retry.md#status-when-max-retries). - Allow updating queue display name, category, max retries, statistics retention, and history retention in the Admin UI/API. - Make it possible to change the sort order of queues and queue categories in the Admin UI. # 2.23.0 _2024-11-24_ - Make it possible to pause jobs after they fail until the error has been resolved. See more [in the docs](../jobs/index.md#pausing-on-error). - Reset retry count when updating a single message in the Admin UI. - Enable editing job category, historical run retention and if a job should be considered degraded when disabled in the Admin UI/API. # 2.22.0 _2024-11-22_ - Make it possible to delete by status in the Admin UI/API - Reset retry count when changing status of a queue item - Respect low MaxNumberOfMessagesPerRun values in non-batched queue jobs - Make it possible to auto cancel jobs after max duration - Allow setting max duration and auto cancel for a job in the AdminUI/API # 2.21.0 _2024-11-18_ - Only filter on status being Pending when leasing messages to make it slightly faster - Remove queue message id database index for queues that doesn't have id - Remove queue database index no longer used - Fix issue with unclosed db connections in memory queue fallback storage - Support authorization in the API and Admin UI, see more [in the docs](../api/index.md#authorization) - Add error handling for unexpected errors in admin UI # 2.20.1 _2024-11-16_ - Only wrap JSON string in quotes if not already wrapped # 2.20.0 _2024-11-16_ - Make it possible to invoke arbitrary Nexus Functions in the Admin UI (see [docs](../nexus-functions/index.md#invoking-functions-in-the-admin-ui) # 2.19.2 _2024-11-11_ - Fix issue where CommerceMind.Nexus.Azure package initialization didn't return the fluent Nexus initialization # 2.19.1 _2024-11-11_ - Pass topic/queue name to the connection string and subscription name loaders in Azure Service Bus config # 2.19.0 _2024-11-11_ - Make it possible to automatically retry errors in queues through Admin UI and code - Fixes issue that reset retry count to 0 when reaching max retry count - Support fetching from Azure Service Bus Topics as well as Azure Service Bus Queues # 2.18.0 _2024-11-09_ - Make it possible for a queue to be [in-memory](../queues/in-memory.md) - Extract SQL `COUNT(*)` into separate query since it's faster for large data sets # 2.17.0 _2024-10-26_ - Optimize handling of queue messages without id # 2.16.0 _2024-10-21_ - Set the same data type in queue message import temp table as the source table to avoid bad query plan - Make it possible to filter to only see jobs with issues in the admin UI - Group jobs list by category first instead of job type first and category second in admin UI # 2.15.0 _2024-10-20_ - Add more retries to SQL queries for functions - Don't load job progress when it's not needed - Handle SQL Server concurrent update for memory optimized tables error - Delay creating job progress until the first progress is reported - Make enqueueing of messages without id faster - Enqueue messages one by one when the batch is small since it's faster # 2.14.0 _2024-10-12_ - Store latest job run details in main jobs table to avoid joins in hot query - Disable lock escalation in SQL Server to avoid use of rowlock hint - Add retries when fetching historical job runs # 2.13.0 _2024-10-02_ - Make unbatched queue processing as fast as batched - Don't use NOLOCK hint for SQL Server when using read committed snapshot # 2.12.0 - Support calling private methods in functions - Prevent long job result text to break the job details UI - Add support for middlewares for functions - Wrap creation and population of temp tables during enqueueing in an OTel transaction so the performance of it can be measured # 2.11.0 - Add retries to query that runs as part of message enqueueing - Release batch messages that haven't been processed or failed - Show in admin UI if any functions failed to load instead of just hiding them # 2.10.5 - Rewrite sql query so that an index can be correctly used # 2.10.4 - Fix regression that didn't respect the global max retry count for functions # 2.10.3 - Fix issue with using messages as records with batch jobs # 2.10.2 - Await function batches to avoid getting into a state where new functions are started before existing ones finish # 2.10.1 - Add missing database index for deleting processed queue messages - Ensure that next runs in UI and API are always in the future - Ensure that parameters passed to JobResults is of the correct type # 2.10.0 - Improve the layout of job list when you have many jobs split into many categories # 2.9.0 - Fix styling for lists and paragraphs in job markdown description - Fix health check migrations for SQLite - Make it possible to mass update queue item priority in admin UI # 2.8.6 - Add retries when reading job meta data - Keep retrying migrations on transient errors to avoid crashing the application - Lower log levels on warnings that are meant for debugging # 2.8.5 - Fix failing migration for health checks # 2.8.4 - Changes the category of Nexus log messages for queue jobs to make it easier to filter out # 2.8.3 - Fix for supporting queue messages to be records rather than classes # 2.8.2 - Separate short description and description of a job and show short description in listings and description on the details page in the admin UI # 2.8.1 - Fix issue with Nexus functions with arguments not getting called correctly # 2.8.0 - Adds description text for jobs in the admin UI which can be edited in the UI, markdown supported - Adds cancellation token support for Nexus functions - Fixes admin UI to not say "Starts in in X seconds" with a duplicate "in" - Fixes admin UI job start dates to not show dates in past time # 2.7.2 - Fix incorrect use of service scope # 2.7.1 - Use service provider scope in all places where a scheduled job instance is used # 2.7.0 - Changes jobs and functions to use a service provider [scope](https://learn.microsoft.com/en-us/dotnet/core/extensions/dependency-injection#scoped) when creating service instances in order to support services registered with a scoped lifetime. # 2.6.2 - Fix issue with using enums as arguments to Nexus functions # 2.6.1 - Correctly report outcome on Open Telemetry activities/traces # 2.6.0 - Expose metrics and telemetry through [Open Telemetry](https://opentelemetry.io/docs/languages/net/) # 2.5.1 - Fix how casts of Nexus function arguments are handled # 2.5.0 - Adds more filtering and sorting to job runs in admin UI - Fixes bug with serializing GUIDs with `System.Text.Json` # 2.4.0 - Adds filtering of job runs in admin UI - Adds missing database index for job runs # 2.3.0 - Support List.Contains() in IQueueItemUpdater.UpdateStatusWhereAsync() - The health checks system now has a watch dog that will reset any check that is stuck in a running state. Eg if the process crashed after the check finished but before Nexus was able to update the status. - Fix issue where enabling queue history for all queues included queues where messages don't have id - Ensure that the ILoggerProvider for job logs aren't registered multiple times # 2.2.4 - Improve error handling with instance events # 2.2.3 - Add transient database error retries on all methods on `IQueueReader` - No longer assumes that a temp table still exists since it might get deleted when a connection is reopened # 2.2.2 - Fix issue with Postgres and SQL Server that caused an error if you called `IEnqueuer.EnqueueAsync()` with an empty list of messages - Reset migrations if they're stuck in a running state because of a server crash # 2.2.1 - Explicitly drop temp tables instead of relying on connection close to delete them # 2.2.0 - Make it possible to change priority on a message when scheduling a future processing - Improve handling of merging partial messages in SQL Server - Add retries on transient database errors in more situations # 2.1.1 - Fix issue with queue messages that has propertiy types without parameterless constructors # 2.1.0 - Fix issue using `MaxRetries` in `[NexusFunction]` attributes - Fix issue causing JSON serialization errors in the Slack healty reporter - Support enqueueing partial messages, [read more here](../queues/index.md#partial-messages) - Add retries on SQL queries with SQLite that caused "database is locked" errors - Run retries on transient database errors more granularly and in more places # 2.0.0 This release contains a few breaking changes that are described in [the upgrade guide](./upgrade-guide.md). The breaking changes are mostly a code and project reorganization that should be easy to upgrade to. Besides the reorganization some notable changes are: - The scheduled jobs system now has a watch dog that will reset any job that is stuck in a running state. Eg if the process crashed after the job finished but before Nexus was able to update the job status. - All NuGet dependencies have been updated to the latest version as well as .NET. - Serilog is no longer a dependency to Nexus. - Newtonsoft.Json is no longer a direct dependency to Nexus, the support for Newtonsoft.Json has been extracted to a separate package. - It's now possible to register [middlewares around job execution](../jobs/index.md#job-middlewares). - It's now possible to set job and log history [per job](../jobs/index.md#log-retention). # 1.23.0 - Allows queue jobs to set custom statuses for messages. [Read more here](../queues/jobs.md#returning-a-custom-status). # 1.22.0 - Adds message priority for queues. [Read more here](../queues/priority.md). # 1.21.1 - Adds feature detection for if queue history is enabled to avoid SQL queries running against tables that doesn't exist # 1.21.0 - Adds opt-in message history for queues with identity. [Read more here](../queues/history.md). # 1.20.3 - Adds retries to the database maintenance job when using Postgres. # 1.20.2 - Fixes risks for deadlocks when job progress hasn't finished writing before the job is done. When a job finishes and cleanup attempts to delete job progress before the job progress has finished writing a deadlock could occur because the update and cleanup/delete tries to write the same row. # 1.20.1 - Removes a query hint when using SQL Server that required READ COMMITTED isolation level. The recommended isolation level is READ COMMITTED SNAPSHOT, but is not a hard requirement. # 1.20.0 - Makes it possible to initiate a soft graceful shutdown which means that all jobs, functions, etc that has already started will run until finished but no new work will be started. See more [here](/general/hosting). # 1.19.0 - Fixes regression around running internal Nexus migrations. # 1.19.0 - Makes it possible to pass a transaction to `DbConnectionScope`. # 1.18.0 - Makes it possible to stop queue processing when an error occur, and pause it until no messages in the queue has an `Error` or `AwaitingRetry` status any more. See more [here](/queues/jobs#pausing-on-error). # 1.17.0 - Adds `IDbConnectionScope` which lets you pass an existing db connection object to Nexus. See more [here](/general/database#controlling-the-connection). # 1.16.1 - Fixes an issue where extra job runs (such as a retry) disappeared if they needed to aquire a mutex and that mutex was already taken # 1.16.0 - `IQueueItemUpdater.DeleteAndReturnAllAsync()` added as a safe way to read and delete all messages in a queue in a single and safe operation - `IScheduledJobMetaDataRepository.GetJobMetaDataAsync()` added # 1.15.6 - Let outcome descriptions line break in admin UI # 1.15.4 - Expire the jobs meta data cache after five seconds to prevent stale cache issues # 1.15.3 - Add debug logging to queue processing jobs # 1.15.2 - Use `long` for queue message ids everywhere - Ensure that changing a job schedule doesn't immediately trigger a new run if the schedule is moved forward in time # 1.15.1 - Fix issue causing `MaxNumberOfMessagesPerRun` not to work on queue jobs # 1.15.0 - Make it possible to control how many messages a single queue job run is allowed to process - Show the Nexus version in the admin UI - Don't show a Disable button for a health check for a disabled job # 1.14.4 - Fix issuing causing some health checks to not show up in admin UI # 1.14.3 - Fix issuing making it impossible to resolve a failed job run # 1.14.2 - Fix links to job details in health check details - Add caching for health check statistics to avoid polling the database on the health overview - Health periods are now not filled out to include the space in time between two periods # 1.14.1 - Change job health check to not generate a unique result description every time it's called # 1.14.0 - Stores health check results so it's possible to see the history of a health check in the Admin UI # 1.13.1 - Fix issue which caused migrations to be run out-of-order # 1.13.0 - Make it possible to use Nexus job logs in the admin UI without Serilog. Previously you needed to use Serilog in order for your job logs to get the correct context/scope values attached to know which job they originated from. Nexus now uses standard .NET logging for this which means that you no longer need to use Serilog for it. See more in the documentation under Jobs / Logs. # 1.12.1 - Reset healthchecks when an instance is starting. If an instance crashed or was deployed in the middle of when a healthcheck was running it can still be reported as running after the deploy or restart finishes. This fixes that all healthchecks are reset that claims to be running on the instance that is just starting up. # 1.12.0 - Added `JobResults.CompletedWithWarnings()` which lets a job return a successful status but have the run flagged with a warning sign in the Admin UI and the job health check status become `Degraded` # 1.11.0 - Messages that are `Leased` will now be transitioned into a new status called `Abandoned` if the lease expires due to a server crash. See [docs about abandoned messages](../queues/index.md#abandoned-messages). - Messages that are scheduled for a retry now has a new status called `AwaitingRetry` instead of being in the `Error` status. - A new max retry limit has been introduced which defaults to 10. If the same message has been retried more times than that the message is transitioned to the `Error` status and won't be automatically retried again. - Instance id and correlation id is now set on queue messages when they are leased to be able to find logs for the job that processed a message. # 1.10.1 - Minor styling issue fixed in Functions list in Admin UI # 1.10.0 - Make it possible to group queues in the Admin UI by category just like what's possible for jobs # 1.9.0 - Fix issue where the admin UI could display "0" instead of "No logs found" - Make it possible to have a new job be initially disabled # 1.8.2 - Fix issue which could cause an infinite loop in `ScheduledJobExecutorService` when using multiple servers and jobs that executes frequently - Lowered log level from information to debug on some logs that occur frequently # 1.8.1 - Includes debug symbols in new `snupkg` packages, published to the Github NuGet feed # 1.8.0 - Removed unnecessary database calls that were made to release job mutexes, even for jobs without any mutexes - Nexus will now react slightly quicker to the cancellation token that signals that the app is shutting down - A new way of dealing with exceptions in background services ensures that Nexus is more resilient against intermittent database failures # 1.7.0 - Made it possible to customize the example message shown in the admin UI when enqueueing a new message manually. - `IScheduledBatchQueueJob.GetBatchSize()` has been deprecated in favor of `IScheduledBatchQueueJob.BatchSize`. The `GetBatchSize()` method will be removed in the next major version. # 1.6.8 - Add missing indexes to mutex table to avoid deadlocks when attempting to release job mutexes # 1.6.7 - Apply less locking when fetching job data from SQL Server to avoid deadlocks # 1.6.6 - Fix issue in API usage of Nexus Functions with SQL Server # 1.6.5 - Fix concurrency issue with leasing queue messages and marking functions as running when using Postgres. # 1.6.4 - Fix issue where instance heartbeat healthchecks reported the time in 12 hour - Fix text overflow in health check admin UI for long descriptions # 1.6.3 - Dispose `CancellationTokenSource` in `ScheduledJobExecutorService` after use # 1.6.2 - Add horizontal scrollbar for long queue processing log in admin UI # v1.6.1 - Remove testing code in the admin UI that slipped through in 1.6.0 # v1.6.0 - Make it possible for jobs to report their progress and display it in the admin UI - Use Tailwind instead of CSS Modules in admin UI # v1.5.4 - Use higher query timeout for all queries in IDatabaseMaintenanceService - Show resolved/dismissed state for latest run in job list # v1.5.3 - Fix issue with database polling instance events # v1.5.2 - Fix issue with admin UI not rendering correctly when not having a trailing slash in the url (ie just /admin and not /admin/) # v1.5.1 - Fix issue with duration incorrectly being stored as decimal in SQLite when integer is expected # v1.5.0 - Make it possible to mark a failed job run as resolved or dismissed with an optional markdown comment - Make it possible to add a comment to queue messages - Store both latest job run and latest meaningful job run to since some jobs run very frequently but doesn't perform work all the time - Fix when cancelling a queue job the historical run status becomes should be Cancelled rather than Completed # v1.4.0 - Add RetryLater as a successful job outcome # v1.3.2 - Fix issue with database maintenance on SQL Express # v1.3.1 - Reset any health checks marked as running when an instance is starting # v1.3.0 - Ask for a connection string every time the database is opened to make it possible to use connection strings with updated credentials # v1.2.1 - Add query timeout of 5 minutes for database maintenance queries - Make it possible to update queue message status by filtering on message JSON properties - Increase timeout when deleting old job runs and logs # v1.2.0 - Show a health check for a disabled job as disabled - Implement graceful shutdown which attempts to stop Nexus background processes before the .NET application begins to terminate to ensure that all jobs etc aren't running when the application terminates - Generate a unique Nexus instance id for Azure App Service where a single VM can run multiple applications # v1.1.0 - Increase query timeout for updating queue messages to correct status - Make it possible to update all queue items to a certain status - Make it possible to mark a queue item as processed and schedule another processing of the message in the future - Only show main menu items in the admin UI if they have been configured - Add WITH (HOLDLOCK) to all SQL Server MERGE statements to avoid insert conflicts # v1.0.0 Initial stable release. --- # Rollback from v3 to v2 The scripts below reverse the built-in Nexus 3.0 database upgrade for PostgreSQL and SQL Server. **Nexus automatically captures a v2 rollback checkpoint during the first upgrade**, before queue initialization or regular migrations change the database. You do not need to remember to run a separate capture script. The rollback restores the v2 schema and saved settings while retaining messages and job history written since the upgrade. A database rollback cannot undo work already processed, external side effects, or broker changes. To restore the exact pre-upgrade database contents, restore the full v2 backup instead. Coordinate that restore with application data and brokers so that already processed work is not unintentionally repeated. ## Download the scripts | Database | Script | Settings at the top of the file | | -------- | ------ | ------------------------------- | | PostgreSQL | [v3-to-v2-postgres.sql](/upgrade/v3-to-v2-postgres.sql) | `target_schema` (default `public`) | | SQL Server | [v3-to-v2-sqlserver.sql](/upgrade/v3-to-v2-sqlserver.sql) | `@TargetSchema` (default `dbo`) | These files only perform rollback, using the checkpoint Nexus created internally during the upgrade. Set the schema, then run the whole file as one script against the intended database. Do not execute selected statements. For SQL Server, run it as one batch, outside an existing transaction. The rollback script requires database compatibility level 130 or later; automatic checkpoint capture during upgrade does not impose that requirement. The scripts target the built-in v3.0 migrations and an installation whose Nexus framework and queue tables reside in one schema. Set the schema explicitly; PostgreSQL's search path does not determine the rollback scope. Separate Nexus databases need separate checkpoints and rehearsals. Include memory-queue fallback databases in the rollback plan too; a fallback-only database without the framework migration/key-value tables needs an adapted script or backup restoration. Do not use these files unchanged for a later release that adds migrations, an application-specific database provider, or an installation split across schemas. Review the additional changes first. ## Prepare before upgrading 1. [Deploy and run the latest v2](upgrade-guide.md#deploy-the-latest-v2-first), and let database and queue initialization finish everywhere. The scripts cannot establish which package version was actually deployed. 2. Rehearse the upgrade and rollback against a disposable copy of that database, using the intended v2 and v3 application artifacts. Include representative persisted messages, parameters, filters, and broker integrations. 3. At cutover, stop all Nexus producers and workers, including separately hosted producers and memory-queue hosts. Let in-flight processing finish or resolve its leases with v2 before upgrading. 4. Take a full v2 database backup and preserve the v2 application artifact, configuration, broker resources, and offsets/backlog recovery plan. Automatic checkpoint capture does not replace these backups. 5. Start v3. Under the database startup lock, Nexus captures `nexus_v2_rollback_checkpoint` or `NexusV2RollbackCheckpoint` before idempotent queue initialization, table renames, or category conversion. Verify the checkpoint and successful initialization before resuming normal traffic. Keep the checkpoint table. Checkpoint capture belongs to Nexus's database initialization; there is no capture script to run or download. If capture fails, the upgrade stops before queue initialization and regular migrations proceed; fix the capture failure before retrying. Fresh v3 installations have no v2 state and do not create a checkpoint. If an upgrade is interrupted after capture, retrying preserves the original checkpoint even if queue/schema changes have already started. Capture does not run on the normal cached startup path. At upgrade, the checkpoint captures settings, category JSON, migration history, and queue schema/index definitions, not the queued messages. During rollback it also archives incompatible health-check results/statistics and v3-only instance events before removing them from active tables. Allow space for that archive, especially when health-check history is large. It is not a substitute for a full backup. Job categories need special handling because v3 replaces `ScheduledJobCategories` with `JobCategories` and serializes the renamed types again. Simply changing the key or replacing type names in JSON does not restore the original type hashes or custom serializer output. The checkpoint preserves that original value exactly. If v3 has already run without a checkpoint, the original values cannot be recovered reliably from the upgraded database. Nexus does not manufacture a v2 checkpoint from v3 state. The rollback script refuses to proceed without its checkpoint, rather than silently losing the original category JSON or other settings. Restore the v2 backup, or prepare and rehearse an installation-specific rollback using the original settings and schema from that backup. Simply installing a build with automatic capture after upgrading cannot recover that earlier state. ## What rollback restores | Change | Rollback action | | ------ | --------------- | | Six renamed job tables | Rename jobs, runs, progress, mutexes, logs, and extra runs back to their `ScheduledJob` names. Existing rows, identity/sequence values, indexes, and permissions remain attached to those tables. | | Queue lease tracking | Remove `leased_version` / `LeasedVersion` and `status_after_release` / `StatusAfterRelease` from existing main queue tables, after checking for unresolved leases. | | Queue lease indexes | Restore the captured v2 `(status, priority, id)` index definition and remove temporary indexes left by an interrupted v3 lease-index migration. Index rebuilding takes locks and needs time, disk space, and transaction-log capacity. | | Activation and processing settings | Drop the queue-activation table and the six new queue metadata columns: activation delay, concurrency per instance, lease size, batch size, processing threads, and message lease time. | | Saved job and queue settings | Restore the captured configuration for existing jobs and queues, including schedules cleared when adopting readiness-driven processing, disabled states, descriptions, default parameters, and queue settings. Changes to those settings made while running v3 are replaced by the checkpoint values. | | Job categories | Restore the original v2 key and serialized value, including its original version and timestamp. Remove the v3 category key. | | Job-run initiators | Map `QueueReadiness` to v2's automatic `Schedule` initiator in current/latest run details, historical runs, and extra runs. Retain the remaining execution details and history. | | Health-check results | Archive results containing the renamed `JobHealthCheckData` type in the checkpoint. Clear their cached result and last-called timestamp so v2 can run the check again. Archive and remove incompatible health-check statistics from active history. Compatible results/statistics and disabled states remain unchanged. | | Database instance events | Archive and remove `QueueMessagesReadyInstanceEvent` and `JobStartRequestedInstanceEvent`, which v2 cannot deserialize. Preserve other events, durable extra runs, and queued messages. These best-effort v3 notifications are not replayed on v2. | | Initialization and migration records | Remove reserved `Nexus.Startup.*` entries and the explicitly listed v3 migration records so a later v3 upgrade runs its migrations again. Preserve the v2 ledger and unrelated key-value entries. | The scripts preserve current queue messages, job-run history, extra runs, next-run parameters, and runtime execution details. Restoring a stale checkpoint of those values could repeat already completed work or lose work added after the upgrade. Review their payload contracts for v2 compatibility before restarting v2. Historical `AwaitingRelease` entries remain in history; v2 displays an unrecognized status as `Unknown`. Health-check and instance-event cleanup recognizes the type metadata written by Nexus's built-in System.Text.Json and Newtonsoft.Json serializers, including custom type-hash salts. It does not rewrite type names or guess hashes. Custom serializers/binders that use another payload format need their own compatibility review. Archived rows use checkpoint kinds `health-result`, `health-statistics`, and `instance-event`; their `value` / `Value` contains the original row, with the serialized payload retained as a string. Keep these rows for inspection; copying them back into v2's active tables would reintroduce the incompatible types. ## Execute a rollback 1. Stop all v3 producers and workers and keep normal traffic paused. Resolve remaining `Leased` and `AwaitingRelease` messages with v3 before reverting; do not blindly convert them to `Pending`, since processing may already have produced side effects or a newer payload may be waiting for release. 2. Back up the current v3 database and relevant broker state too, preserving work written after the upgrade. 3. Confirm the script's database and schema match the checkpoint and run the entire file. It acquires the v3 startup lock and applies its database changes in a transaction. On failure, the changes are rolled back; investigate the reported condition before retrying. 4. Restore the v2 application artifact and configuration, including generated-queue routing, queue-job registrations and transaction behavior. Restore broker topology and direct-producer destinations as needed. SQL does not move messages between old and new Redis keys, Service Bus entities, RabbitMQ resources, or Kafka consumer groups. 5. Start one latest-v2 instance and verify initialization, saved schedules/settings, category loading, representative persisted payloads, queue/job history, refreshed health checks and their history, and database instance-event polling. Review archived health results and control events. Check job state left by the stopped v3 instances. Then start the other instances and resume traffic deliberately. Rollback refuses to continue when it finds an unknown migration, changed queue/filter schemas or custom indexes, added/removed queue resources, changed job/queue metadata identities, conflicting v2/v3 tables, or unresolved queue leases. The scripts do not drop new queues, discard messages, or guess how to reverse application migrations. Prepare a specific inverse for those changes, or use backup restoration. Optional v3 features also need an application plan: distributed processors and new named queues cannot become v2 registrations through SQL. Retention cleanup, deleted messages/history, payload changes, filter backfills that replaced schema/data, and external processing side effects cannot be reconstructed from the settings checkpoint. Keep the checkpoint and backups until the rollback is verified. A completed rollback marks the checkpoint as used and refuses to reuse it for another rollback. On a later v2-to-v3 upgrade, Nexus archives that used checkpoint with a timestamp/suffix and automatically captures a fresh v2 baseline. An unconsumed checkpoint is preserved on retries and restarts. --- # From v2 to v3 Version 3 changes how queue processing is started and adds distributed queue processors and named queues. There are also breaking changes to service lifetimes, public APIs, and database initialization. **Note** that v3 renames some tables and values. Read more in [Database initialization in v3](./upgrade-guide.md#database-initialization-in-v3). This guide covers the changes to review before upgrading. For code replacements, configuration examples, and a validation checklist, use the [detailed upgrade guide](detailed-upgrade-guide.md). That guide is written as a skill for an LLM carrying out the upgrade. ## Deploy the latest v2 first Before upgrading an existing installation to v3, **upgrade to the latest Nexus v2 release, deploy it, and let it complete startup and database initialization in every environment being upgraded**. This applies to installations already running v2 as well as those coming from v1. Updating package references without deploying and running that version is not enough. Make sure every Nexus database and its registered queues have been initialized successfully by the latest v2. If different applications register different queues, run the relevant applications so their initialization also completes. Once this prerequisite is satisfied, stop the v2 instances before deploying v3. V3's faster database initialization adopts the existing v2 schema and then records which initialization work has completed. The supported upgrade path assumes v2's migrations and compatibility work have already finished; do not skip the intermediate deployment from an older version. If the latest v2 is already deployed and initialization has completed, this prerequisite is already satisfied. Direct upgrades from v1 are no longer supported. V3 no longer adds the old v1 queue priority columns or rewrites serialized v1 type and assembly names, so any remaining v1 persisted data must also be processed or migrated through v2. ## .NET and package versions All Nexus packages now require .NET 10. Update the application's target framework, build environment, and hosting runtime before upgrading the Nexus packages together to v3. ## Job and queue API names Schedule-neutral APIs now use `Job` instead of `ScheduledJob`. For example, `AddScheduledJobs()` becomes `AddJobs()`, `[ScheduledJob]` becomes `[Job]`, and `IScheduledJobMetaDataRepository` becomes `IJobMetaDataRepository`. Queue jobs use `IQueueJob` and `IBatchQueueJob` instead of `IScheduledQueueJob` and `IScheduledBatchQueueJob`. Ordinary scheduled jobs still implement `IScheduledJob` as regular jobs always have a schedule. Obsolete APIs have been removed, if your build is failing on v3 because of this the easiest thing is to use the latest v2 and check and fix all build warnings on obsolete APIs. ## Queue jobs can start when messages are ready Queue jobs without a schedule now start when messages become ready. `EnqueueAsync()` requests activation after messages are committed, so replace `EnqueueAndStartProcessingAsync()` with `EnqueueAsync()`. A queue job with a cron schedule remains scheduled-only. Existing schedules are stored in the database, so removing `DefaultSchedule` from the class does not clear an existing schedule. Clear it through the API or Admin UI when you want to switch that queue job to readiness-driven processing. Conversely, a queue job with an empty schedule that previously only ran manually can now start automatically. Review calls to `EnqueueAndStartProcessingAsync()` on scheduled queues: enqueueing no longer brings their next run forward. If an additional run is needed, request it explicitly using the job APIs. See section about `IJobStartRequester`. Database providers register persistent activation storage automatically. Redis activation storage is also available; in-memory activation storage is only suitable for a single instance setup. `ActivationDelay` can collect a burst before processing starts, while `ProcessingDelay` still controls when each message becomes eligible. Queue jobs no longer report automatic progress by default to get better default queue performance. Set `AutomaticProgressMode` to `QueueJobProgressMode.Percentage` to retain the old behavior, or `ProcessedCount` to show a handled count without first counting the queue. Re-enqueueing a leased message in a database or in-memory queue now puts it in `AwaitingRelease`. The current processor keeps its lease, and the latest enqueue takes effect when it finishes. Update custom status handling and monitoring to recognize this as a message still being processed. External pending storage keeps its provider-specific behavior. Previously `AwaitingRelease` wasn't needed as Nexus only ever allowed a single instance to process a queue at any time, which is no longer true in v3. ## Starting jobs with `IJobStartRequester` Immediate parameterless `RequestJobStartAsync()` calls now send a fast, best-effort instance event. Only instances running the continuous job loop handle these requests. They may be coalesced, lost during downtime, or cause successive runs on different instances; one request does not guarantee exactly one run. If a job is running at the same time that `IJobStartRequester` requests it to start another run will start immediately after. The purpose of this is to process anything added at the end of the job run that the job didn't process. Keep frequent immediate start calls on `IJobStartRequester` to benefit from the faster implementation: it avoids extra-run database writes and the associated cache invalidations across instances. Do not routinely replace these calls with scheduled extra runs. Check how the application recovers pending work after a missed signal; having no schedule does not by itself mean frequent calls should become database writes. Use `IJobMetaDataRepository.ScheduleExtraRunAsync()` for specific start requests that must themselves be persisted, or applications using `RunAllJobsPendingSinceLastCallAsync()`, which cannot handle the events. Calls with a non-null parameter object or an explicit delay/start time still schedule extra runs. Applications can now send immediate requests without registering the job locally or calling `AddJobs()`, provided they configure the shared instance-event broker. Use the correct persisted job name, especially for generated jobs or jobs bound to several queues. ## Distributed processors and named queues are optional Your existing scheduled queue jobs can work just like before. [Distributed queue processors](../queues/processors.md) are an alternative when independent messages should be processed concurrently across application instances. They do not provide job runs, schedules, parameters, or mutexes. Each concrete queue can have one queue job or one distributed processor; in-memory queues do not support distributed processors. [Named queues](../queues/named.md) let several queues share one message type. If you adopt them, choose the destination explicitly when enqueueing and preserve existing names where data and metadata should be retained. The new Admin UI can override processing settings per queue, so also review any saved overrides when changing code defaults. This is a good option when you need a dynamic set of queues for the same thing. Perhaps one queue per order type but you don't know at compile time how many order types you have. ## Generated queues need explicit routing Existing generated queues also use the new named queue infrastructure. Nexus no longer creates runtime message or job subclasses, so generated message contracts and job implementations must be concrete. Choose the queue name when enqueueing to a generated queue. For the previous `ShouldEnqueue` routing behavior, configure a shared `RoutingQueueName` and enqueue to that virtual entry point. Type-only enqueueing no longer broadcasts to generated destinations, and direct enqueueing to a concrete destination bypasses `ShouldEnqueue`. Keep the existing concrete queue and generated job names to retain their data and settings. See [generated queues](../queues/generated.md). ## Most Nexus services are now scoped Operational Nexus services, including enqueuers, repositories, functions, serializers, and integration services, are now scoped. Code running in a request or Nexus execution scope can inject them normally. Singleton application services and `BackgroundService` implementations must create a scope for each independent operation instead of injecting scoped Nexus dependencies or resolving them from the root provider. Choose application lifetimes based on their own requirements; they do not all need to become scoped. Keep the scope alive until its asynchronous work finishes, and do not cache a Nexus service beyond that scope. Process-wide services such as the instance-event broker remain singleton. ## Entity Framework queue integration is scope-bound Nexus now uses the configured `DbContext` from the same scope as the enqueuer. Register it as scoped with `AddDbContext`, avoid DbContext pooling, and configure it with `UseNexus(serviceProvider)`. All local enqueue interfaces now respect that unit of work. Database messages commit with entity changes, and in-memory messages become visible after commit. Use asynchronous saves and transaction commits; pending messages are discarded if the context is disposed without saving. Review code that previously relied on an enqueue overload persisting immediately. For independent enqueue operations that should persist without saving a context, use `ImmediateQueueOperationScope`. Nexus's API, MCP, and broker ingress already do this. Creating a new DI scope cannot join the caller's existing transaction. Remote API enqueueing and external pending storage are sent immediately and cannot be rolled back with EF; use a database outbox queue if they must follow the entity transaction. See [Entity Framework integration](../queues/entity-framework.md). ## External pending storage External pending storage now has a separate durable readiness path. RabbitMQ uses a fan-out exchange with processing and readiness queues, Azure Service Bus uses a topic with two subscriptions, and Kafka uses separate processing and readiness consumer groups. Redis keys now include a shared cluster hash tag and a readiness stream. Review provisioning, broker permissions, and any producers writing directly to these resources. Drain or explicitly migrate messages in old resources before switching; database migrations do not move broker data. Redis's old keys and Azure Service Bus's old queue entities are not read by the new topology. RabbitMQ producers must publish to the new exchange instead of directly to the processing queue. See [external pending storage](../queues/external-pending.md#readiness-bridges-and-direct-producers). This is separate from integrations that consume an existing external queue into a Nexus queue. Those keep their configured source, but custom Kafka/RabbitMQ transformers and deserializers may need API changes described in the detailed guide. ## The Admin UI is registered separately from the API `AddApi()` no longer registers the Admin UI. Reference `CommerceMind.Nexus.UI`, call `AddAdminUI()`, and keep `UseNexusAdminUI()`. Set `NexusAdminUIOptions.ApiBaseUrl` explicitly; when hosted together, it should match `NexusApiOptions.BasePath`. Environment name, notices, and other UI options are configured on `NexusAdminUIOptions` instead of through `AddApi()`. The new Admin UI is the default. `UseExperimentalNewVersion` has been replaced by `UseLegacyVersion`, which temporarily enables the legacy UI. A separately hosted UI now defaults to the `nexus` path instead of `/`. Review custom paths and reverse-proxy configuration. See [Admin UI configuration](../admin-ui/index.md). Custom HTTP clients also need updating: - `GET /api/jobs` returns an object containing `jobs` and `categories`; the separate categories endpoint has been removed. - `GET /api/queues` omits counts by default. Request selected queues with `queueNamesWithCounts`, or all counts with `includeCountsForAll=true`. - Historical runs use `correlationId` instead of a database `id`, and extra-run IDs are now 64-bit integers. - Queue creation payloads no longer accept `startProcessing`; queue activation follows the configured processing mode. Regenerate clients against the v3 API, including its nullable queue-job schedule and new message status. ## JSON and other integrations System.Text.Json polymorphic JSON now requires a valid type hash by default. Most applications using the default serializers need no change. Check older persisted data and custom writers that emit `$type` without `$typehash`; if needed, temporarily set `JsonOptions.RequireTypeHash = false` while migrating that data. Keep the existing type-hash salt. Custom instance-event subscriptions now use asynchronous `Subscribe()` handlers and must dispose the returned subscription. The old `OnInstanceEventRecieved` event has been removed. The retired Teams webhook publisher has been removed. Replace it with the Teams bot or outgoing bridge integration. ## Database initialization in v3 After the [latest-v2 deployment and initialization](#deploy-the-latest-v2-first) has completed, stop all v2 instances before starting v3 against the same database. V3 automatically renames job tables, so a rolling deployment with v2 and v3 sharing the database is not supported. Update custom SQL, monitoring, and database permissions that reference the old table names. The first v3 startup completes migrations, adopts existing queue schemas, and rebuilds queue lease indexes. Large queues need additional time, disk space, and database I/O; SQLite index creation also blocks writers. Allow initialization to finish before bringing the application into service. Later starts skip migration-history and schema checks when the stored initialization state is unchanged. New queues and changed filterable properties are still initialized, including backfilling filters from existing messages. Keep those messages readable by the updated types and coordinate instances when making incompatible schema changes. Nexus no longer automatically repairs manually changed tables. Leave Nexus-owned schema and reserved `Nexus.Startup.*` key-value entries intact. SQLite does not coordinate concurrent initializers; initialize it from one process. SQL Server and PostgreSQL coordinate v3 initialization through database locks. Also review `QueueSystemOptions.DisableHistory`: it now overrides history settings saved in the database. ## Rollback to v2 PostgreSQL and SQL Server [rollback scripts](rollback-v3.md) are available to reverse the built-in v3.0 database changes. Nexus automatically captures a v2 checkpoint before queue initialization and regular migrations, including saved schedules and the original serialized job categories. If you want to return to v2, stop all Nexus instances and run the selected rollback script to restore the v2 schema and captured settings. v3 has been thoroughly tested and is stable. Being able to rollback to v2 is just a service we provide in case you want to try out upgrading in a lower environment to test the new features out, and decide if you're ready to migrate to v3 or wait. # From v1 to v2 Version 2 contains a number of breaking changes from version 1. The core logic and concepts all remain the same in Nexus and this release is mostly about code and project reorganization, removing unnecessary dependencies and updating to .NET 8. ### New NuGet package structure Instead of having the different sub systems such as `CommerceMind.Nexus.Jobs` in their own NuGet packages they have been consolidated into the new `CommerceMind.Nexus` package and the abstraction packages have been consolidated into `CommerceMind.Nexus.Abstractions`. This reorganization means that many interfaces have been moved to new namespaces. Eg `IScheduledJob` which now lives in `CommerceMind.Nexus.Abstractions.Jobs` instead of `CommerceMind.Nexus.Jobs.Abstractions`. The interface names are still the same, so it should be easy to either search-and-replace the namespaces or let Visual Studio/Rider/etc guide you to the new names. Another reorganization change in this is that the implementations for the different databases are no longer included in the main packages. Instead they have been extracted into `CommerceMind.Nexus.Postgres`, `CommerceMind.Nexus.SqlServer`, and `CommerceMind.Nexus.Sqlite`. Allowing you to only include the database(s) you want. ### New fluent service registration In v1 there was a number of extension methods on `IServiceCollection` that you had to call in a specific order. In v2 this has changed to a fluent registration on `IServiceCollection` where the order of which method you call is no longer important. This is how the v2 registration looks like: ```cs builder.Services. .AddNexus() .AddSingleServerInstanceEvents() .AddFunctions() .AddScheduledJobs() .AddQueues() .AddSqliteConnection() // Don't forget this one! .Build() ; ``` Note that you can still split up your service registration as long as the call to `.Build()` is done last, as that will register additional services based on your configuration. You can for example do this: ```cs builder.Services.AddNexus().AddSingleServerInstanceEvents(); builder.Services.AddNexus().AddFunctions(); builder.Services.AddNexus().AddScheduledJobs(); builder.Services.AddNexus().AddQueues(); builder.Services.AddNexus().AddSqliteConnection(); builder.Services.AddNexus().Build(); ``` Each of these lines could be in different projects as well, as long as the call to `Build()` happens last. All calls to `builder.Services.AddNexus()` will return the same Nexus registration object for that specific service collection. It's also safe to call any method on the Nexus registration object multiple times. ### Nexus no longer uses Newtonsoft.Json by default Instead `System.Text.Json` is used by default and the Newtonsoft.Json serializers have been extracted into a separate package called `CommerceMind.NewtonsoftJson` which you can use by calling `builder.Services.AddNexus().AddNewtonsoftJson()`. You probably don't need Newtonsoft.Json, this is only if you've modified how the JSON in Nexus is serialized with custom converters. A notable change here is that the JSON formatting rules for ASP.NET will no longer be applied for queue messages in the API. Instead the JSON formatting rules for Nexus is applied, which means that you might get a response looking like this: ```json { "id": 4823771, "status": "Processed", "retryCount": 0, "message": { "Id": null, "SomeProperty": null, "ExampleEnum": "Value1" } } ``` Where the outer object have gotten kebabCase properties but the message object has PascalCase properties. You can control both of these ([see docs about JSON](./json.md)) formatting options. ### Nexus now requires >= .NET 8 v1 supported .NET 6 but v2 now reqires at least .NET 8. The only thing you need to do is to upgrade your .NET version in your application if you haven't already done so. ### Deprecated methods and properties have been removed Methods and properties that have been deprecated in v1 have been removed in v2. If you upgrade to v1.23.0 and fix all build warnings about obsolete methods and properties you'll automatically be prepared for the removals of them in v2. ### Default job history retention is now 30 days Previously the default was to keep job history in the database forever. To avoid having Nexus growing your database indefinitely the default has been changed to 30 days. You can still configure this just like before, but since you can now have different retention per job the property is now called `DefaultHistoricalRunsRetention`: ```cs builder.Services.AddNexus().AddScheduledJobs(options => { options.DefaultHistoricalRunsRetention = TimeSpan.FromDays(10); }); ``` You set it per job with the `[ScheduledJob]` attribute like this: ```cs [ScheduledJob("MyJob", HistoricalRunsRetention = "30.00:00")] public class MyJob : IScheduledJob { ... } ``` ### The IEnqueuer and IEnqueuer interfaces have been simplified Previously `IEnqueuer` inherited from `IEnqueuer` which it no longer does. In v1 every message type was registered as a `IEnqueuer` which means that you could get all enqueuers by requesting `IEnumerable` like this: ```cs public class DynamicEnqueuer(IEnumerable enqueuers) { public Task EnqueueAsync(IQueueMessage message) { var enqueuer = enqueuers.Single(e => e.MessageType == message.GetType()); return enqueuer.EnqueueAsync(message); } } ``` In v2 there is only a single non-generic `IEnqueuer` registered which does the above for you. Meaning that you can pass a `IQueueMessage` message to it and it will figure out which concrete `IEnqueuer` to pass it on to. Both the `IEnqueuer` and `IEnqueuer` interfaces have been simplified to take a message and an `EnqueueContext` instead of having multiple different overloads for when you want to specify a custom status, a date for when to process it, etc. So when you did this in v1: ```cs await enqueuer.EnqueueAsync(message, DateTime.UtcNow.AddHours(1)); ``` To enqueue a message to be processed in one hour you now do this: ```cs await enqueuer.EnqueueAsync(message, new EnqueueContext { ProcessAfter = TimeSpan.FromHours(1) }); ``` ### No longer using Serilog Nexus previously used Serilog to capture logs happening inside job runs. With v2 it instead has a [`ILoggerProvider`](https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.logging.iloggerprovider?view=net-8.0) which is registered by the scheduled job system. In v2 you can use any logging framework you want as long as it writes to external logging providers. Note that this is turned off by default in Serilog and that you need to enable it, eg like this: ```cs builder.Host.UseSerilog((context, services, configuration) => { configuration.ReadFrom.Configuration(context.Configuration); // This one is important }, writeToProviders: true); ``` If you've used the jobs logging in v2 you also need to remove the Nexus Serilog sink from your configuration: ```json "Serilog": { "Using": ["CommerceMind.Nexus.Jobs"], "WriteTo": [{ "Name": "NexusJobLogs" }] } ``` ### Changes to the `IJobLogProvider` interface For consistencys sake the `IJobLogProvider` interface has been renamed to `IJobLogReader` since there's also a `IJobLogWriter` interface. And instead of just getting a date to fetch logs newer than Nexus now passes in a date range. You also get the log levels passed as `LogLevel` enums rather than strings. Also note that `JobLogEvent.Level` has changed data type from `string` to the `LogLevel` enum. --- # Healthchecks Nexus integrates with and extends [Health checks in ASP.NET Core](https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/health-checks?view=aspnetcore-6.0). All functions, jobs, and queues automatically get health checks that signals their health. The only thing you need to do in an ASP.NET Core application is call this in your `Program.cs`: ```cs app.MapHealthChecks("/health"); ``` The jobs, queues and functions subsystems in Nexus initializes the health checks subsystem as part of their initialization so you don't have to do that explicitly. But if you want to change any of the default options you can initialize it directly like this: ```cs builder.Services.AddNexus().AddHealthChecks(options => { // Set this to true if you're not using ASP.NET. Then Nexus will periodically call the health checks for you // and publish a health report just like in ASP.NET. options.UseHealthCheckBackgroundService = false; // If you're not using ASP.NET you can configure how often the background service calls the health checks. options.HealthCheckBackgroundServicePollDelay = TimeSpan.FromSeconds(30); // How long Nexus should keep results of health checks to display historical status in the admin UI options.StatisticsRetention = TimeSpan.FromDays(30); // If you have health checks definied in other assemblies you can add them here to get Nexus to find them options.AssembliesToScanForHealthChecks.Add(typeof(MyType).Assembly); // This lets you dynamically exclude any health check types that you don't want Nexus to register options.HealthCheckFilter = healthCheckType => !healthCheckType.Name.Contains("HealthCheckIDontWant"); }); ``` You can see the health checks included in the demo environment here:\ [https://commerce-mind-nexus-a9bsbwgth3cgfnde.swedencentral-01.azurewebsites.net/admin/healthchecks](https://commerce-mind-nexus-a9bsbwgth3cgfnde.swedencentral-01.azurewebsites.net/admin/healthchecks) ## Teams and Slack integrations If you wish to get Teams or Slack notifications when health checks are failing you can use the [Teams integration](./teams.md) or [Slack integration](./slack.md) for Nexus. ## When are health checks invoked? If you're using ASP.NET the health checks will be called when anyone is pinging the `/health` endpoint and when you access the health checks in the admin UI. If you've registered a `IHealthCheckPublisher` that will also cause the health checks to be called an additional two times per minute by ASP.NET. When you create a custom health check as described below you can choose to have a scheduled or unscheduled health check. If you create a scheduled health check Nexus will ensure that your health check is never executed more often than the schedule allows. Nexus will return the previous result for such a health check until it's time to call it again. ## When not using ASP.NET If you're not using ASP.NET you can still use the Nexus health checks. You should initialize the health check system like this: ```cs builder.Services.AddNexus().AddHealthChecks(options => { options.UseHealthCheckBackgroundService = true; options.HealthCheckBackgroundServicePollDelay = TimeSpan.FromSeconds(30); }); ``` You should also create your own [`IHealthCheckPublisher`](https://docs.microsoft.com/en-us/dotnet/api/microsoft.extensions.diagnostics.healthchecks.ihealthcheckpublisher?view=dotnet-plat-ext-6.0) and reigster it with the `IServiceCollection` like this: ```cs using CommerceMind.Nexus.Abstractions.Core.DependencyInjection; builder.Services.AddScopedHealthCheckPublisher(); ``` The Nexus background service calls publishers at `HealthCheckBackgroundServicePollDelay` (30 seconds by default). The adapter retains the publisher in one scope until disposal, so it can keep state between reports. Store the last reported status to avoid repeating unchanged notifications. If the publisher needs a fresh context per report, create an inner operation scope. A publisher that has only singleton dependencies can still use a direct singleton registration. ## Scoped dependencies Nexus registers custom checks as scoped. They can inject scoped repositories and application services; do not resolve them from the root provider or capture them in a singleton publisher. See [dependency injection](../general/dependency-injection.md). `JobsOptions.ScheduleTimeZone` controls job cron schedules; scheduled health checks continue to evaluate their own schedule in UTC. External pending storage also contributes readiness-bridge health checks; see [queue monitoring](../general/monitoring.md#queue-readiness-and-distributed-workers). ## Adding more health checks There's two health check interfaces included. One that runs on demand, and one that is scheduled. ### `INexusHealthCheck` Use this implementation for checks that you always want to run on demand. Their results are never cached but should then respond quickly and not be very costly. For example: ```cs public class ExampleHealthCheck : INexusHealthCheck { public string Name => "MyHealthCheck"; // This needs to be unique public string DisplayName => "Health check running a lot"; public string? Description => "A longer description of what the health check does"; // You can return null if you don't need a description public string? Category => "My health checks category"; // Used to group checks in the Admin UI public IEnumerable Tags => new string[] { }; public string? AdminUILinkUrl => null; public async Task CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default) { return HealthCheckResult.Healthy($"Everything is fine!"); } } ``` ### `INexusScheduledHealthCheck` In some cases you want a health check to run on a schedule rather than firing every time someone pings the health status endpoint. For this you can use the `INexusScheduledHealthCheck` interface which much like jobs has a cron schedule. When the service executes the health checks it will only execute a scheduled check if enough time has passed since the last time it was called. Otherwise the system will return the previous result for that check. Another benefit of `INexusScheduledHealthCheck` is that the `CheckHealthAsync()` method is passed the previous result of that check. This means that you can determine health based on a previous run. An example: ```cs public class ExampleScheduledHealthCheck : INexusScheduledHealthCheck { public string Schedule => "@every_minute"; // See: https://github.com/HangfireIO/Cronos#macro public string Name => "MyHealthCheck"; // This needs to be unique public string DisplayName => "Health check running every minute"; public string? Description => "A longer description of what the health check does"; // You can return null if you don't need a description public string? Category => "My health checks category"; // Used to group checks in the Admin UI public IEnumerable Tags => new string[] { }; public string? AdminUILinkUrl => null; public async Task CheckHealthAsync(NexusHealthCheckResult? previousHealthCheckResult, ScheduledNexusHealthCheckInitiator initiator, HealthCheckContext context, CancellationToken cancellationToken) { var previousCount = (int?)previousHealthCheckResult?.Result.Data?["count"]; var data = new Dictionary(); var currentCount = GetCurrentCount(); data["count"] = currentCount; if (currentCount - previousCount < 10) { return HealthCheckResult.Unhealthy($"Everything is NOT fine!", null, data); } return HealthCheckResult.Healthy($"Everything is fine!"); } private int GetCurrentCount() { // Look up value } } ``` Here we're using the `Data` property on `HealthCheckResult` to store any properties we need in the next run to determine the health. This can be used to see if counters such as the amount of `Pending` messages in a certain queue increases too much. #### Schedule The `Schedule` property is a cron expression just like [`DefaultSchedule`](../jobs/index.md#ischeduledjobdefaultschedule) on jobs, and it can be any expression supported by [Cronos](https://github.com/HangfireIO/Cronos). The above example uses one of the supported [macros](https://github.com/HangfireIO/Cronos#macro) in Cronos. Note that there's a difference between the `DefaultSchedule` property on `IScheduledJob` and the `Schedule` property on `INexusScheduledHealthCheck`. The default schedule on a job is only read by Nexus the first time the job is deployed and registered in the database. Using eg `CronSchedule.EveryMinute()` for the job will generate a random second (eg `16`) for the second during in a minute that the job will start on. This means that you can have many jobs using `CronSchedule.EveryMinute()` but they won't start at exactly the same time. Instead they will start on random seconds to spread out the load they generate. The health checks `Schedule` property on the other hand are read every time Nexus checks if it's time to invoke them. So if you use `CronSchedule.EveryMinute()` for a health check it'll sometimes execute the check multiple times in a minute and sometimes less frequent than a minute. Eg if the clock is `12:00:10` and the schedule says `9 *, *, *, *, *` the check is executed. And if the next time the schedule is evaluated it becomes `19 *, *, *, *, *` it means that it'll only be 10 seconds between the times the check is called. If you still want to generate a random second for a health check you can generate it when the application starts and then keep returning that cron schedule. Like this: ```cs public class ExampleScheduledHealthCheck : INexusScheduledHealthCheck { private static string _schedule = CronSchedule.EveryMinute(); public string Schedule => _schedule; } ``` This will make sure that the schedule is stable during the time that the application lives and only generate a new schedule when the application is restarted or a new version is deployed. ## Registering custom health checks If you created an implementation of `INexusHealthCheck` or `INexusScheduledHealthCheck` it becomes automatically registered if the type exists in your entry/application assembly. If your checks exists in a different assembly you can instruct Nexus which assemblies to scan: ```cs builder.Services.AddNexus().AddHealthChecks(options => { options.AssembliesToScanForHealthChecks.Add(typeof(MyHealthCheck).Assembly); }); ``` --- # Slack integration If you're using Slack you can use the `CommerceMind.Nexus.Slack` NuGet package to get realtime updates when jobs and queues are failing. The package contains a [`IHealthCheckPublisher`](https://docs.microsoft.com/en-us/dotnet/api/microsoft.extensions.diagnostics.healthchecks.ihealthcheckpublisher?view=dotnet-plat-ext-6.0) implementation that publishes [health check failures](./index.md) to Slack. ## Notifications Queue errors:  "Show errors" button clicked:  Errors resolved:  Job errors:  "Show error logs" button clicked:  ## Installation The first thing you need to do is to create a Slack app for your organization. Go to [https://api.slack.com/apps](https://api.slack.com/apps) and click `Create New App`. You can use the following manifest to create the app from: ```yml display_information: name: Nexus description: Info notices from Nexus background_color: "#18113f" features: bot_user: display_name: Nexus always_online: false oauth_config: scopes: bot: - chat:write.public - chat:write settings: interactivity: is_enabled: true org_deploy_enabled: false socket_mode_enabled: true token_rotation_enabled: false ``` If you want to you can use our logo for the app icon: [nexus.png](./img/nexus.png) After you've created the app you need to install the `CommerceMind.Nexus.Slack` NuGet package and configure it in your `Program.cs` like this in the application that runs background jobs: ```cs builder.Services.AddNexus().AddSlackPublisher(options => { options.AccessToken = "xoxb-XXX"; options.AppToken = "xapp-XXX"; options.ChannelId = "CXXX"; options.AdminUIUrl = "https://url-to-nexus-adminui.com"; }); ``` You should only enable the Slack publisher in the application that runs background functions and/or jobs. The Slack publisher can handle multiple instances/servers and will correctly handle instances starting and stopping, but if you run the publisher in an application that doesn't run jobs you won't get notified when jobs are failing. You can find your Slack app access token in the left menu item OAuth & Permissions on the page for your app ([https://api.slack.com/apps/](https://api.slack.com/apps/)). The value starts with `xoxb`. The app token can be found under Settings / Basic Information -> App-Level Tokens. The value starts with `xapp`. The channel id can be found by opening the channel/DM/group chat you want Nexus to post messages to and click on the name at the top. In the bottom of the About tab you find the channel id. Note that the channel id is not visible in the Slack mobile app. The admin UI url doesn't have to be publically available. The Nexus Slack app use it to generate links that a Slack user can click on. All communication between Slack and Nexus is done over web sockets so your Nexus application doesn't have to be publically accessible. --- # Health check statistics The result of all Nexus health checks are stored when they go from one health status to another or when the description returned from the check changes. This means that if you click on a health check in the Admin UI you get statistics about when the health check status changed and for how long the period was when it was unhealthy or degraded. The built-in checks for Nexus queues and jobs for example will show a history of when jobs fail or when queues have messages with error status and for how long. The health check dashboard also shows indicators for each check of which status the previous results for the check was. This makes it easier to spot that a check that is currently healthy recently had an unhealthy period. ## Importance of the result description Nexus uses the description text of a health check result to determine that something changed about the check. For the health check statistics every time the result description changes while the status is unhealthy or degraded the result is stored. And if the status is healthy and the description changes a counter is incremented to signal how many different healthy checks there's been in a period. This means that you should consider what you include in the description. You should not include values such as the current date/time because that will change every time the check is called. Avoid doing this: ```cs public async Task CheckHealthAsync(NexusHealthCheckResult? previousHealthCheckResult, ScheduledNexusHealthCheckInitiator initiator, HealthCheckContext context, CancellationToken cancellationToken) { return HealthCheckResult.Healthy($"Everything was fine on {DateTime.UtcNow}"); } ``` But you should be doing things like this: ```cs public async Task CheckHealthAsync(NexusHealthCheckResult? previousHealthCheckResult, ScheduledNexusHealthCheckInitiator initiator, HealthCheckContext context, CancellationToken cancellationToken) { var errorCount = GetErrorCount(); if (errorCount == 0) { return HealthCheckResult.Healthy($"No errors found"); } else { return HealthCheckResult.Unhealthy($"{errorCount} errors found"); } } ``` By including the error count in the description Nexus will store the result if the error count changes and you'll get more details when you look at the unhealthy period. If you want to include even more details you should use the data dictionary on `HealthCheckResult` like this: ```cs var data = new Dictionary(); data.Add("Something", "Interesting"); return HealthCheckResult.Unhealthy($"{errorCount} errors found", null, data); ``` The data dictionary will be serialized and stored and displayed in the health check details. ## Statistics retention By default Nexus will store the health check statistics for 30 days before it's deleted. You can set a different value on the options object when initializing Nexus health checks: ```cs builder.Services.AddNexus().AddHealthChecks(options => { options.StatisticsRetention = TimeSpan.FromDays(7); }); ``` If you set this to `TimeSpan.Zero` then Nexus will never store any statistics for health checks. --- # Microsoft Teams integration If you're using Teams you can use the `CommerceMind.Nexus.Teams` NuGet package to get realtime updates when jobs, queues, and health checks are failing. The package contains an [`IHealthCheckPublisher`](https://docs.microsoft.com/en-us/dotnet/api/microsoft.extensions.diagnostics.healthchecks.ihealthcheckpublisher?view=dotnet-plat-ext-8.0) implementation that publishes [health check failures](./index.md) to Teams through a Teams bot. Nexus v3 has removed the old Teams incoming-webhook publisher. Use direct Teams bot mode or outgoing-only bridge mode as described below. ## Register a Teams bot If you haven't already you need to register a Teams bot. This guide helps you with the steps: https://learn.microsoft.com/en-us/microsoftteams/platform/teams-sdk/get-started/quickstart-register?pivots=csharp Note that when registering your bot you will need a public url into Nexus that Teams can communicate with. So read the rest of the docs first before registering your bot if you're unsure about the url. The guide helps you register a Teams bot in your Teams workspace and will create a `Teams` section in appSettings.json with `ClientId`, `ClientSecret`, `TenantId` which you'll need regardless if you use the bridge mode or direct mode. If you use the direct mode you'll add these to the appSettings.json of your Nexus application. If you use the bridge mode you'll add them to the appSettings.json file of your bridge app instance. Note that regardless if you use direct mode or bridge mode Teams needs a public url that it can ping you on. In direct mode that's directly to your Nexus instance and in bridge mode it's a separate, public facing web application that acts as a bridge between your Nexus application and Teams. ## Notifications The Teams bot publisher stores the conversation where the bot is installed and sends proactive Adaptive Card messages to that chat or channel. It can update the original card when a failing check changes, when someone clicks a bot-side button, and when the check becomes healthy again. Supported actions include: - Failing scheduled job: show or reload error logs, and open the job in the Nexus UI. - Long-running scheduled job: show or reload logs, and open the job in the Nexus UI. - Queue errors: show or reload the latest 5 errors, and open the queue in the Nexus UI. - Generic health check: show the health check status and open the health checks page in the Nexus UI. Nexus keeps the active Teams message IDs in its key-value store, so cards can still be updated after the application restarts. In multi-instance Nexus deployments, only one Nexus instance publishes Teams health messages at a time. ## Direct mode Use direct mode when the Nexus application that publishes health checks is reachable by Teams over public HTTPS. Install the NuGet package `CommerceMind.Nexus.Teams` and configure the Teams bot publisher in `Program.cs`: Configure the Teams bot publisher in `Program.cs`: ```cs builder.Services .AddNexus() .AddTeamsBotPublisher(options => { // A required secret that protects linking a Teams conversation to notifications. Use a long random value. options.LinkCode = builder.Configuration["NexusTeamsBot:LinkCode"]!; // This is optional and used to add link buttons to the messages to open the Nexus UI. options.AdminUIUrl = "https://url-to-nexus-adminui.com"; // The default is Degraded. Set to Unhealthy to skip degraded health checks. options.MinimumHealthStatus = HealthStatus.Degraded; }); var app = builder.Build(); app.UseNexusTeamsBot(); ``` Teams commonly uses `/api/messages` in Bot Framework examples, but the path itself is not a Teams requirement. `UseNexusTeamsBot()` registers the Teams bot endpoint at `/nexus-teams/bot/messages` by default but if you need another route, pass it explicitly: ```cs app.UseNexusTeamsBot("api/messages"); ``` Create and configure a Teams app/bot using the standard Microsoft Teams SDK configuration. The application that runs Nexus must be reachable by Teams, and the bot messaging endpoint should point to your Nexus application, for example: ```text https://your-nexus-host.example.com/nexus-teams/bot/messages ``` If `AdminUIUrl` is configured, add that host to the Teams app manifest `validDomains` list. Teams requires this for `Action.OpenUrl` buttons in Adaptive Cards. Linking a conversation requires the configured `LinkCode`, so that being able to install or message the bot is not by itself enough to subscribe a conversation to health alerts. ### Link conversation Installing the bot does not auto-link the conversation; send `@name-of-bot link {linkCode}` to link the current Teams conversation, and send `@name-of-bot unlink` from the same conversation to remove it. Multiple chats or channels can be linked, and health notifications are sent to every linked conversation. Keep the link code out of shared channels where possible, and remove the linking message after the link has been created. ## Outgoing-only bridge mode Use bridge mode when Nexus is hosted as an internal application that Teams cannot reach over the internet. The public bridge receives Teams traffic, while the internal Nexus application only makes outbound HTTPS requests to the bridge. Install `CommerceMind.Nexus.Teams` in the internal Nexus application and configure bridge publishing: ```cs builder.Services .AddNexus() .AddTeamsBridgePublisher(options => { options.BridgeUrl = new Uri("https://teams-bridge.example.com"); options.NexusAppId = "erp-prod"; options.ApiKey = builder.Configuration["NexusTeamsBridge:ApiKey"]!; }); ``` The internal Nexus application does not need Teams bot credentials and does not need to call `UseNexusTeamsBot()`. Host a public bridge application with `CommerceMind.Nexus.Teams.Bridge`: ```cs // Program.cs builder.Services.AddNexusTeamsBridge(); builder.Services.AddNexusTeamsBridgeSqlServer(builder.Configuration.GetConnectionString("NexusTeamsBridge")!); // or AddNexusTeamsBridgePostgres(...) // or AddNexusTeamsBridgeSqlite(...) var app = builder.Build(); app.UseNexusTeamsBridge(); ``` The bridge is just a simple ASP.NET Core web application with at least the `CommerceMind.Nexus.Teams.Bridge` package installed that you need to deploy to somewhere that Teams can access. It needs a stable IP or domain that you can use when you register your teams bot. The bridge needs a database to maintain state between restarts of the bridge application. It uses Entity Framework internally. You can point it to an existing database, but you should not point it to a database already owned by EF. The bridge registers the Teams bot endpoint at `/nexus-teams/bot/messages`. Configure the Teams app/bot messaging endpoint to point to the public bridge: ```text https://teams-bridge.example.com/nexus-teams/bot/messages ``` A single bridge can serve many Nexus applications. The bridge application is designed to run as one active server, with support for short blue-green deployment overlap. ## Bridge database schema The bridge uses EF Core migrations for its own tables. `AddNexusTeamsBridge()` runs `Database.MigrateAsync()` at startup by default, so a new bridge database is created automatically when the configured database user has schema permissions. Set `AutoMigrateDatabase = false` when migrations are applied by a deployment pipeline, migration bundle, or another operational process. ## Bridge app registration Nexus app registrations are managed directly in the bridge database. There is intentionally no bridge admin HTTP API. Insert one row per Nexus application: ```sql INSERT INTO NexusTeamsBridgeApps ( NexusAppId, DisplayName, AdminUIUrl, ApiKey, LinkCode, IsEnabled, CreatedAtUtc, UpdatedAtUtc ) VALUES ( 'erp-prod', 'ERP Production', 'https://internal-erp-nexus/admin', 'generated-long-random-secret', 'generated-long-random-link-code', 1, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP ); ``` API keys and link codes are stored in plain text in the bridge database. Protect the bridge database accordingly, use long random values, and rotate them by directly updating the `ApiKey` and `LinkCode` columns. The `ApiKey` authenticates the internal Nexus application to the bridge (server-to-server). The `LinkCode` is a separate secret that protects who may subscribe a Teams conversation to an app's notifications: because the `NexusAppId` is not secret and is easy to guess, linking additionally requires the link code so that being able to message the bot is not enough to receive an app's health alerts. Keep the link code out of shared channels where possible; because it is typed into a Teams message it is visible in that conversation's history, so rotate it if it is exposed. Delete the link message after the link is created. The default Nexus-to-bridge authentication uses the `X-Nexus-Teams-Bridge-Api-Key` HTTP header. This keeps bridge API credentials separate from the Teams SDK's own `Authorization` handling. Internal Nexus applications can replace request signing by registering `INexusTeamsBridgeRequestSigner` before calling `AddTeamsBridgePublisher()`. Bridge hosts can replace request authentication by registering `INexusTeamsBridgeRequestAuthenticator` before calling `AddNexusTeamsBridge()`. ### Link conversation Link a Teams conversation to a registered Nexus app by sending one of these messages to the bridge bot, using the app's configured link code: ```text @name-of-bot link erp-prod {linkCode} ``` The link code must match the `LinkCode` stored for that app or the link is refused. Running the same command from another chat or channel adds that conversation to the app's notification destinations; the bridge sends each health notification to every linked conversation for that Nexus app. Remove the current Teams conversation from a Nexus app with: ```text @name-of-bot unlink erp-prod ``` --- # Introduction Commerce Mind Nexus is a .NET 10+ based library and solution from [Commerce Mind](https://www.commercemind.se) for running scheduled background functions and jobs, and populating and processing queues of messages. The solution is built primarily with e-commerce data flows in mind but it's a general solution that works outside of e-commerce as well. ## Key features - Distributed as NuGet packages, [Apache 2.0 licensed](tou.md) - Over 600 integration and unit tests to ensure stability - Execute a class method with retries in the background using [Nexus functions](./nexus-functions/index.md) - Schedule [jobs](./jobs/index.md) to run every X seconds/minutes/hours/etc - Out-of-schedule [job runs](./jobs/index.md#extra-job-runs) - Scale out and run jobs and functions on multiple servers - SQL based queues with [repository](https://martinfowler.com/eaaCatalog/repository.html) like access - Extreme throughput [when pure SQL based queues aren't enough](./queues/external-pending.md) - [In-memory](./queues/in-memory.md) queues with fallback storage when performance requires it - Versioned queue messages with optional identity to ensure safety - Access to [previous version of messages](./queues/index.md#previous-message-version) to enable sophisticated change tracking - [Virtual queues](./queues/virtual.md) with one or more backing queues - Process queue messages in [batch or one by one](./queues/jobs.md), starting [when messages are ready](./queues/activation.md) or on a cron schedule - [Distributed queue processors](./queues/processors.md) with workers on every instance - [Named queues](./queues/named.md) sharing one message contract - [Per-queue processing settings](./queues/processing-settings.md) editable without redeploying - Full [HTTP API](./api/index.md) and [Admin UI](./admin-ui/index.md) - Enqueue over [HTTP API](./queues/enqueueing.md) to let other services create messages - Great introspection, debuggability, and visibility into the contents of the queues - Easily re-enqueue failed messages - [Monitoring](./general/monitoring.md) of failed jobs, queue messages, and functions - Custom statuses for queue messages to allow grouping and saving for later - Enqueueing messages for [processing in the future](./queues/index.md#scheduling-messages-for-future-processing) - Optionally [keep processed messages](./queues/retention.md) - Retry processing a queue message in X minutes - Scheduled [health checks](./general/monitoring.md) - Can easily be embedded into an existing application - No external dependencies required, everything can be run locally on a dev machine - Metrics exposed with Open Telemetry - Out-of-the-box integrations with [Azure](./queues/azure.md), [Teams](./healthchecks/teams.md), [Slack](./healthchecks/slack.md), and [RabbitMQ](./queues/rabbitmq.md) See the section about [Nexus vs FaaS](./faas.md) if you read this and thought "Well I already have Azure Functions for this". If you've used [Hangfire](https://www.hangfire.io/) before you might want to read the [Hangfire vs Nexus](./general/hangfire-vs-nexus.md) comparison. ## Demo environment There's a demo environment that's deployed to a free tier Azure Web App here: **Admin UI**\ [https://commerce-mind-nexus-a9bsbwgth3cgfnde.swedencentral-01.azurewebsites.net/admin/](https://commerce-mind-nexus-a9bsbwgth3cgfnde.swedencentral-01.azurewebsites.net/admin/) **API**\ [https://commerce-mind-nexus-a9bsbwgth3cgfnde.swedencentral-01.azurewebsites.net/swagger/](https://commerce-mind-nexus-a9bsbwgth3cgfnde.swedencentral-01.azurewebsites.net/swagger/) You can click on any buttons you like, start any job or function or do anything with the queues to get a feel for what the system does. :::note[Demo environment] Since the demo environment is using a free tier Azure Web App the scheduler is paused when nobody is using the API or admin UI which means that the jobs won't run exactly according to schedule. ::: ## Getting started The first thing you need to do is to get access to the NuGet packages which are hosted on a Github packages. Just talk to us and we'll add you to the repo and give you access to the NuGet feed! Once you've gotten a Personal Access Token for the feed you need to configure NuGet to look for packages in this feed:\ [https://nuget.pkg.github.com/Commerce-Mind/index.json](https://nuget.pkg.github.com/Commerce-Mind/index.json) You use your Github username as username and the Personal Access Token as password. Read more about the Github NuGet feed here:\ [https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-nuget-registry](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-nuget-registry) When you have access you need to install these NuGet packages in a .NET 10 or later project: ``` CommerceMind.Nexus # One or more of: CommerceMind.Nexus.Postgres CommerceMind.Nexus.SqlServer CommerceMind.Nexus.Sqlite ``` The API and Admin UI are separate packages for ASP.NET Core hosts. Install both if you want the bundled UI: ``` CommerceMind.Nexus.Api CommerceMind.Nexus.UI ``` When you've installed the packages you open your `Program.cs` and add this initialization code: ```cs var builder = WebApplication.CreateBuilder(args); builder.Services.AddSingleton(); builder.Services.AddControllers(); builder.Services .AddNexus() .AddSingleServerInstanceEvents() .AddFunctions() .AddJobs() .AddQueues() .AddApi() .AddAdminUI(options => options.ApiBaseUrl = "/") .AddSqliteConnection() // Or: // .AddPostgresConnection() // .AddSqlServerConnection() // Don't forget to call this, as this is what finializes the Nexus registration and validates it! .Build() ; var app = builder.Build(); // If you want the admin UI app.UseNexusAdminUI(); app.UseHttpsRedirection(); app.UseAuthorization(); app.MapControllers(); app.Run(); ``` The example service above is stateless and can be singleton. Services that inject Nexus dependencies should follow the [scoped dependency guidance](./general/dependency-injection.md). Now let's create the most simple functions, jobs and queues possible, make sure to place it in the same .NET project as your `Program.cs` file. ```cs public class ExampleService { public void ExampleMethod(int arg) { Console.WriteLine("ExampleMethod was called with arg: " + arg); } } public class ExampleJob : IScheduledJob { private readonly IEnqueuer _enqueuer; private readonly INexusFunction _nexusFunction; public ExampleJob(IEnqueuer enqueuer, INexusFunction nexusFunction) { _enqueuer = enqueuer; _nexusFunction = nexusFunction; } public string DefaultSchedule => CronSchedule.TimesPerMinute(10); public async Task ExecuteAsync(CancellationToken cancellationToken) { await _enqueuer.EnqueueAsync(new ExampleQueueMessage()); await _nexusFunction.RunInBackgroundAsync(x => x.ExampleMethod(123)); return JobResults.Completed("This went great!"); } } public class ExampleQueueMessage : IQueueMessage { } public class ExampleQueueProcessingJob : IQueueJob { public async Task ProcessMessageAsync(ExampleQueueMessage message, CancellationToken cancellationToken) { return ProcessResults.Completed("The message was processed!"); } } ``` Place a breakpoint in the `ExampleMethod()`, `ExecuteAsync()` and `ProcessMessageAsync()` methods and start the application. The breakpoints should get hit within a couple of seconds. Now we have a very basic example of one job scheduling a function as well as producing messages and placing them in a queue, and another job that process the messages. If using a web application you'll also be able to access the API on eg `/api/jobs` and `/api/jobs/ExampleJob/historical-runs`. The Admin UI is available on `/nexus`. With the above setup you're using SQLite as the database which is great for quickly getting started, but it's not really fit for a heavy production load. Read more about database setup in the [database section](./general/database.md). The queue job has no schedule, so enqueueing a ready message is enough to start it. A cron schedule makes a queue job scheduled-only. Register `AddQueueProcessors()` instead of a queue job when you want distributed processing across instances. When you've got the above working you should dig deeper into the docs here to understand everything you can do with the system! ## Trusted by Nexus is used in production by some of the Nordic region's most well-known e-commerce companies. ### Lyko Lyko is a leading Nordic beauty and hair care retailer operating across Sweden, Norway, Finland, and several other European markets. With a complex, constantly evolving IT environment spanning multiple group companies, Lyko manages B2C flows through third-party logistics providers using a custom-built Order Management System. Nexus is integrated directly into that OMS, giving developers immediate visibility into logs and making it easy to pinpoint and resolve problems across a distributed infrastructure. The flexible adapter concept makes it straightforward to add new integrations rapidly as Lyko expands into new markets and channels. > "We have complex flows spanning several companies and systems, and Nexus lets us build new integrations quickly without losing control. The visual, accessible UI also means the whole team, not just the developers, can act when something does not look right. > > Nexus solves our needs very well and is now an established component in our blueprint for developing new products and services where we apply the Transactional Outbox integration pattern." > > — Emil Svebeus, CTO Lyko ### Cervera Cervera is one of Sweden's most established kitchenware and home decor retailers, currently undergoing a major IT transformation — relocating their IT environment, replacing their ERP, and reworking the integrations that connect it all. Through Nexus, Cervera has unified their order flows across ERP, billing, purchasing, and analytical systems into a single, well-monitored integration layer. A key outcome has been shifting day-to-day monitoring from the technical team to business personnel, with live health check dashboards displayed on office screens. This gives the whole organisation fast, clear visibility into whether integrations are running as expected — and makes it easy to respond when they are not. > "Commerce Mind Nexus is a stable product that works well handling our integrations. What stands out is the visual overview and the ability to easily receive alerts about problems and rerun jobs with a single click." > > — Rasmus Andersson, CTO Cervera ### Kjell & Company Kjell & Company is Sweden's leading specialist retailer in consumer electronics, cables, and accessories, with more than 80 stores across the Nordic region alongside a growing e-commerce presence. Running an omnichannel operation means keeping a large number of systems — storefronts, warehouse management, ERP, and supplier feeds — continuously in sync. Nexus handles the background job orchestration and queue processing that keeps order and inventory data flowing reliably between those systems, both in real time and on schedule. The monitoring capabilities help the development team surface and fix integration issues quickly, before they reach customers. ### Thule Thule is a globally recognised Swedish brand specialising in outdoor lifestyle products — roof racks, bike carriers, bags, and luggage — sold in more than 100 countries. Running e-commerce operations at global scale demands tight, dependable integration between online storefronts, ERP systems, and an international network of logistics partners. Nexus manages the scheduled background jobs and queue processing that keep orders, product data, and fulfilment information flowing accurately across Thule's digital infrastructure. The solution's stability and built-in observability are a natural fit for a company where reliability is non-negotiable. ### Lekia Lekia is one of Sweden's most beloved toy retail chains, with stores across the country offering a wide range of toys, games, and children's products. Behind the scenes, keeping stock, orders, and supplier data in sync across both physical and digital channels requires a robust integration layer. Nexus manages the background job processing and queue handling that ensures Lekia's order flows run reliably, whatever the volume. The health check and monitoring features give the team confidence that everything is working as it should — and a clear path to action when something needs attention. ### Nordic Nest Nordic Nest is one of Europe's leading online destinations for Scandinavian design, founded in 2002 and offering more than 250 curated brands — from iconic names like Georg Jensen and Marimekko to contemporary favourites like Ferm Living and Normann Copenhagen. Operating across multiple European markets with worldwide shipping, Nordic Nest has built a reputation not only for their exceptional product selection but for staying at the forefront of e-commerce technology. Their engineering team embraces modern tooling and infrastructure to deliver a seamless shopping experience at scale, making them one of the most technically ambitious retailers in the Nordics. Nexus fits naturally into that culture — providing reliable, observable background processing that keeps their order and data flows running without friction. ### Ark Ark is Norway's largest bookstore chain, with more than 150 stores across the country alongside a thriving e-commerce platform at ark.no and a dedicated reading app for digital books. Owned by Gyldendal ASA, Ark has built one of the most ambitious omnichannel retail operations in Scandinavia, seamlessly connecting physical stores, online orders, and click-and-collect services for millions of customers. The scale of their operations — handling millions of book titles and a high volume of daily orders across every channel — demands an integration layer that is both robust and highly observable. Nexus provides the background job orchestration and queue processing that keeps Ark's order flows, inventory data, and system integrations running reliably across their entire operation. ### Hatstore Hatstore is a Swedish e-commerce company founded in 2011 and today one of Europe's leading online retailers for headwear, with over 20,000 caps, hats, and beanies from more than 100 brands including New Era, '47, and Mitchell & Ness. Operating regional webshops across more than ten European markets — including Sweden, Germany, the UK, France, Spain, and the Netherlands — Hatstore manages a complex multi-market setup with localised storefronts and a shared logistics infrastructure. Beyond off-the-shelf products, Hatstore also offers a custom design service with embroidery and printing, adding another layer of operational complexity to their order flows. Nexus helps keep those flows — across markets, product types, and fulfilment paths — running smoothly with full visibility into what is happening at every step. ### Science Fiction Bokhandeln Science Fiction Bokhandeln has been Sweden's home for science fiction, fantasy, and horror since 1984 — selling books, comics, manga, board games, role-playing games, and collectibles across stores in Stockholm, Gothenburg, Malmö, and Linköping, as well as online at sfbok.se. With a deeply passionate customer base and a catalogue spanning tens of thousands of titles and products, keeping inventory, orders, and supplier data in sync across both physical and digital channels is no small task. Nexus handles the background job processing and queue management that keeps their order flows running reliably, giving the team clear visibility into their integrations and the ability to act fast when something needs attention. --- # Jobs An ordinary scheduled job is a C# class that implements [`IScheduledJob`](https://github.com/Commerce-Mind-Nexus/Nexus/blob/main/src/job-engine/Nexus.Abstractions/Jobs/IScheduledJob.cs). [Queue jobs](../queues/jobs.md) use `IQueueJob` or `IBatchQueueJob` and can start when messages are ready instead of using a schedule. Both are registered through `AddJobs()`. [Distributed queue processors](../queues/processors.md) are a separate alternative without job runs. The simplest possible implementation of a job would look like this: ```cs public class MyJob : IScheduledJob { public string DefaultSchedule => CronSchedule.TimesPerMinute(10); public async Task ExecuteAsync(CancellationToken cancellationToken) { Console.WriteLine("The job ran"); return JobResults.Completed(); } } ``` By default the name used as the key for the job in the database is its type/class name. You can use the `[Job]` attribute to give it a stable name and display name: ```cs [Job("myjob", DisplayName = "My really great job")] public class MyJob : IScheduledJob { } ``` Now we'll use `myjob` as the key in the database and the admin UI will display it as `My really great job`. Now you can also rename the job class without loosing the meta data. ## IScheduledJob.DefaultSchedule This property tells the job engine what the default schedule of the job is, and in the above example we're scheduling it to run ten times per minute. But `DefaultSchedule` is just a string that should contain a valid cron schedule (including seconds) so you can return any valid cron expression. `CronSchedule` here is just a helper class for generating such schedules. The engine is using [`Cronos`](https://github.com/HangfireIO/Cronos) to parse cron expressions, so any expression that is valid in `Cronos` will work. The reason for the property being called `DefaultSchedule` rather than `Schedule` is because it will only be read once by the system and that is the first time the engine registers the job. After that the schedule is saved to the database and can be changed through the UI and the API. If you set `DefaultSchedule` to an empty string (or use `CronSchedule.NotScheduled()`) the job will get registered but the engine will never execute it unless you start it manually through the UI or the API. ## Schedule timezone Job cron schedules are evaluated in UTC by default. To run jobs at a fixed local time, configure `ScheduleTimeZone`: ```cs builder.Services.AddNexus().AddJobs(options => { options.ScheduleTimeZone = TimeZoneInfo.FindSystemTimeZoneById("Europe/Stockholm"); }); ``` With this setting, `0 0 9 * * *` runs at 09:00 in Stockholm, corresponding to 08:00 UTC in winter and 07:00 UTC in summer. `ScheduleTimeZone` accepts a `TimeZoneInfo` object. Use `TimeZoneInfo.Utc` to preserve the default behavior, `TimeZoneInfo.Local` for the server's timezone, or resolve a named timezone with `TimeZoneInfo.FindSystemTimeZoneById`. Named IDs must be supported on the host; invalid IDs throw when that lookup is called during configuration. You can also supply a custom `TimeZoneInfo`. The setting applies to all job cron schedules, including schedules changed through the UI or API. Configure the same effective timezone on every instance that runs these jobs; `TimeZoneInfo.Local` requires matching server timezone settings. Stored timestamps and API dates, including extra runs, remain UTC. New jobs still run immediately the first time they are registered. Daylight-saving transitions follow [Cronos's behavior](https://github.com/HangfireIO/Cronos#daylight-saving-time): a daily time in the spring clock gap runs at the next valid local time, and a daily time in the autumn clock overlap runs once. ## IScheduledJob.ExecuteAsync() This method is where you'll place the code that should be performed by the job. The example just contains a simple `Console.WriteLine()` which won't be very useful in a real scenario. The job instance is created inside a dependency injection scope, so it can request scoped application and Nexus services through constructor injection. See [dependency injection](../general/dependency-injection.md). :::note[Instantiation] Note that the job might be instantiated for other reasons than to call `ExecuteAsync()` so don't do any work in the constructor, you should only set instance fields. ::: ## CancellationToken The `ExecuteAsync()` method is passed a [`CancellationToken`](https://docs.microsoft.com/en-us/dotnet/api/system.threading.cancellationtoken?view=net-6.0) by the engine which the job should respect. If you've never worked with cancellation tokens before it's a way for the outside to signal into the job that someone wants to cancel the job execution. It's up to every job to respect the cancellation token passed in and if you don't do that the job can't be cancelled. It'll either always run to completion or be forcefully killed if the job service is killed. The cancellation token passed in is a combination of the general service cancellation token and a cancellation token for that specific job run. The service cancellation token can be triggered by a deploy or a machine restart whereas the cancellation token for a specific job is only triggered if someone manually cancels the job through the UI or the API. The difference here is not something a job needs to think about, it only have to concern itself with the cancellation token passed to `ExecuteAsync()`. Here's an example of how you would use the cancellation token: ```cs public async Task ExecuteAsync(CancellationToken cancellationToken) { foreach (var item in await _service.GetBigListOfThingsAsync()) { cancellationToken.ThrowIfCancellationRequested(); // Do stuff with the item } return JobResults.Completed(); } ``` You add calls to `cancellationToken.ThrowIfCancellationRequested();` inside loops or between chunks of code that are time consuming. A general rule is that the job should never execute code for more than one second before calling `cancellationToken.ThrowIfCancellationRequested();` again. `ThrowIfCancellationRequested()` will throw an `OperationCanceledException` which the job engine will catch and report that job run as cancelled. ## JobResults The return type of a job is [`JobResults`](https://github.com/Commerce-Mind-Nexus/Nexus/blob/main/src/job-engine/Nexus.Abstractions/Jobs/JobResults.cs) which is a class that represents the outcome of a job run. `JobResults` contains static methods such as `Completed()` that tells the engine if the job ran successfully. The `Completed()` method takes an optional string describing the outcome of the job which is visible under the historical runs of the job in the UI. Something like `"Exported 5 orders, 2 of which failed"`. ### JobResults.NothingToDo() If your job starts up and you see that there's no work to do, you should return `JobResults.NothingToDo()` rather than `JobResults.Completed()`. Returning `NothingToDo()` signals to the engine that this run isn't interesting, so it won't get saved to the historical runs. This helps to not clutter up the job history in the UI with runs that didn't perform any interesting work. ### JobResults.RetryLater() Sometimes your job detects that it can't do what it's supposed to do right now. It might be that an external system is down or that the data it needs isn't available right now. For jobs that run less frequently this can be an annoying thing. Lets say that you schedule a job to run every night but external systems might have downtime during the night so now you have to wait a whole day for it to run again. Instead of waiting that whole day your job can return eg `JobResults.RetryLater(TimeSpan.FromMinutes(10))` to tell the engine to schedule an extra run of that job in ten minutes. More on extra runs down below. ### JobResults.Failed() `JobResults` contains a method called `Failed()` which you don't need to call manually. Nexus will catch any exceptions that occur and mark the run as failed. This method is useful if you want to return parameters to the next run of the job. Read more about [jobs with parameters here](parameters.md). ### JobResults.CompletedWithWarnings() Sometimes your job does what it's supposed to do but detects that some things aren't really what they should. Something might be taking a longer time than expected, or some import or exports succeeds but generates warnings. In these cases you can return `JobResults.CompletedWithWarnings()` and pass a descriptive text of the warnings. The jobs health checks state will now be `Degraded` (rather than `Healthy` or `Unhealthy`) and will show with an orange warning sign in the Admin UI. You should use standard logging inside your job with additional details about the warnings to be able to resolve it by logging at the job logs. The recommendation is to use this for things that needs to be looked into but isn't urgent or critical enough to use `JobResults.Failed()`. Don't use it for things that aren't actionable since it'll only create noise. ## Max job duration Most jobs have an average expected duration and can sometimes take a much longer time to complete than expected. For some jobs this doesn't matter and the duration can vary a lot. Such as a queue job that sometimes needs to process a lot of messages. But if you have jobs where you want to be notified if the job runs for too long you can set the max duration either in the [Admin UI](../admin-ui/index.md) under the More button in the job details, or by setting `MaxDuration` on the `[Job]` attribute like this: ```cs [Job("myjob", MaxDuration = "00:10")] public class MyJob : IScheduledJob { ... } ``` The value should be any string that can be passed to [`TimeSpan.Parse()`](https://docs.microsoft.com/en-us/dotnet/api/system.timespan.parse?view=net-6.0). In the above example the max duration is set to 10 minutes. When a job has been running for a longer time than `MaxDuration` the [Admin UI](../admin-ui/index.md) will show a warning message about the jobs duration, and the [health check](../general/monitoring.md) for the job will report a `Degraded` health status. Or be cancelled if automatic cancellation is enabled. Read more about automatic cancellation below. ## Automatic cancellation Sometimes you might have jobs that aren't allowed to run for a longer period of time than the defined max duration. In such cases you can enable automatic cancellation either in the [Admin UI](../admin-ui/index.md) under the More button in the job details, or through the job attribute like this: ```cs [Job("myjob", MaxDuration = "00:10", AutoCancel = true)] public class MyJob : IScheduledJob { ... } ``` In the above case the jobs cancellation token will be triggered after ten minutes. ## Disabling a job A job can be disabled using the [Admin UI](../admin-ui/index.md) or the API. This means that the job won't start on it's schedule, or when queue processing is requested. It will only start if you manually start it through the UI or API. This is very useful for when external systems that the job depends on are having issues, or when the job isn't behaving as it should. In order to not forget about enabling important jobs again you can say that the health check for the job should get a `Degraded` status when it's disabled: ```cs [Job("myjob", DegradedWhenDisabled = true)] public class MyJob : IScheduledJob { ... } ``` You can also create a new job directly in a disabled state. This can be useful when the job does senstive work where you need to be in control of when it starts. Any new job will have its first run directly when its deployed for the first time which may not be ideal in all cases. You can set the job to be initially disabled like this: ```cs [Job("myjob", InitiallyDisabled = true)] public class MyJob : IScheduledJob { ... } ``` ## Pausing on error Sometimes a job can be very sensitive to errors and in those cases you might want to prevent a job from running again until the error has been resolved. Nexus lets you handle this by setting `PauseOnError` in the `[Job]` attribute like this: ```cs [Job("myjob", PauseOnError = true)] public class MyJob : IScheduledJob { ... } ``` Previously this could be done by implementing a `bool StopProcessingOnError { get; }` on queue jobs. That property was removed in Nexus 3, and Nexus refuses to start if a queue job still has it. Note that this can be overriden in the Admin UI/API under the More button in the job details page. ## How to register jobs The job engine discovers concrete `IJobBase` implementations, including queue jobs and ordinary scheduled jobs implementing the [`IScheduledJob`](https://github.com/Commerce-Mind-Nexus/Nexus/blob/main/src/job-engine/Nexus.Abstractions/Jobs/IScheduledJob.cs) or [`IScheduledJob`](https://github.com/Commerce-Mind-Nexus/Nexus/blob/main/src/job-engine/Nexus.Abstractions/Jobs/IScheduledJob.cs) interfaces and automatically register them. This is only done automatically for the entry assembly so if you have multiple .NET projects that contains job classes you need to register the assemblies with the service. Call `AddNexus().AddJobs()` to scan additional assemblies for jobs: Repeated calls share `JobsOptions` and accumulate discovered jobs. Libraries should append assemblies and `GeneratedJobs` definitions to the existing lists. Call `Build()` once after all libraries have contributed: queue jobs then bind to the complete queue set, including queues contributed after discovery. Each generated job's queue must be declared before its `AddJobs()` call. See [library contributions](../queues/named.md#contributions-from-libraries) for ordering and conflict rules. ```cs builder.Services.AddNexus().AddJobs(options => { options.AssembliesToScanForJobs.Add(typeof(MyJob).Assembly); }); ``` If an assembly contains jobs you do not want to register, use `JobFilter`: ```cs builder.Services.AddNexus().AddJobs(options => { options.JobFilter = jobType => jobType != typeof(JobThatIDontWant); }); ``` ## Log retention By default the job history is stored for 30 days but you can configure for how long job history and [logs](./logs.md) should be stored like this: ```cs builder.Services.AddNexus().AddJobs(options => { options.DefaultHistoricalRunsRetention = TimeSpan.FromDays(10); }); ``` You can also specify this per job with the `[Job]` attribute like this: ```cs [Job("MyJob", HistoricalRunsRetention = "30.00:00")] public class MyJob : IScheduledJob { ... } ``` The string you set is any string that can be passed to `TimeSpan.Parse()` or the special string `forever` to indicate that Nexus should never delete the job history. Note that this value can be updated in the Admin UI under the More button on the job details page. ## Extra job runs A job typically starts either by its schedule saying that it should start or by someone manually starting the job. But a job can also have extra runs outside of it's normal schedule. An extra run can be scheduled either using the API endpoint `POST /api/jobs/{jobName}/start?startAt=2022-08-19T12:00:00` (date should be in UTC) or using `IJobMetaDataRepository.ScheduleExtraRunAsync()`. Extra runs are persisted in the job database and claimed across processing instances. Use them when a start must survive until a processing instance is available, especially for jobs without a schedule: ```cs await jobMetaDataRepository.ScheduleExtraRunAsync(nameof(MyJob), TimeSpan.Zero, JobRunInitiator.Manual); ``` ## Starting a job programatically Use `IJobStartRequester` for a fast, best-effort trigger, for example to bring forward the next run of a job that already has a schedule. Use `IJobMetaDataRepository.ScheduleExtraRunAsync()` or the API start endpoint with an explicit UTC `startAt` query parameter for a durable extra run. An immediate API start without parameters can use the same best-effort event path. When you request a start without parameters or a start time, the request is sent as an instance event. Only instances running the continuous job loop handle it. Delivery and exactly one run are not guaranteed: requests may be coalesced, lost while processing instances are unavailable or restarting, or cause successive runs on different instances. A running instance records a request to rerun the job after its current run completes, coalescing further requests received during that run. The normal protection against concurrent runs still applies. Calls with a non-null parameter object or an explicit delay/start time currently schedule a durable extra run in the job database. Passing null parameters without a start time still uses the best-effort event path. Call the job repository directly when durable scheduling is required so that the intent is explicit. `IJobStartRequester` doesn't require the job to be registered in the application that requests the start, and it's available as soon as an instance event broker is configured. An API-only application that doesn't call `AddJobs()` can request job starts that are handled by the processing instances, as long as they share instance events. Without `AddJobs()` only starts without parameters and start time are supported, since the other overloads schedule an extra run in the job database. In that case it's up to the requesting application to use the correct job names. `RequestJobStartAsync()` uses the name from `[Job("...")]` or the class name when the job isn't registered locally, so use the overloads that take a job name for queue jobs that are bound to several queues and for generated jobs. ## Job parallelism and mutexes The engine guarantees that a single job never executes multiple times in parallel. Even if it's scheduled to run for eg ten times per minute it will only run once per minute if it takes 60 seconds to complete. Only one instance is allowed to have the job running at any given time. This is done because many jobs communicate with external data sources and having the same job doing that in parallel can often cause bugs. For queue jobs, `NumberOfProcessingThreads` enables parallelism within the active run. Use a [distributed queue processor](../queues/processors.md) instead when every instance should process independent messages concurrently. If you have two or more jobs that you want to ensure never run at the same time you can use the [`[RequireJobMutex]`](https://github.com/Commerce-Mind-Nexus/Nexus/blob/main/src/job-engine/Nexus.Abstractions/Jobs/Attributes/RequireJobMutexAttribute.cs) attribute on your job. Here's an example: ```cs [RequireJobMutex("SomeMutex")] [RequireJobMutex("SomeOtherMutex")] public class Job1 : IScheduledJob { // Code omitted for brevity } [RequireJobMutex("SomeMutex")] public class Job2 : IScheduledJob { // Code omitted for brevity } [RequireJobMutex("SomeOtherMutex")] public class Job3 : IScheduledJob { // Code omitted for brevity } ``` In this example `Job1` can never run at the same time as `Job2` and `Job3`. `Job2` can run at the same time as `Job3` but not as `Job1` since `Job2` and `Job3` needs different mutexes. For application resources shared by otherwise independent jobs, you can acquire mutexes dynamically during execution. V3 allows one processing job per concrete queue; use separate named/generated queues when work needs independent filters and processing settings: ```cs public class MyJob(IJobMetaDataRepository jobMetaDataRepository) : IScheduledJob { public string DefaultSchedule => CronSchedule.NotScheduled(); public async Task ExecuteAsync(CancellationToken cancellationToken) { await using var _ = await jobMetaDataRepository.AcquireMutexesAsync(["some-mutex-name"], cancellationToken: cancellationToken); // The mutex has been acquired return JobResults.Completed(); } } ``` The mutexes are decentralized and safe to use with multiple Nexus servers/instances running your jobs. If you're only using a single server you're probably better of using `SemaphoreSlim` to handle locking. ## Job middlewares In some cases you want to wrap the execution of a job. You might want to add additional log context properties or instrument the job execution in some way. To achieve this you can register one or more `IJobMiddleware` instances in the service collection. Eg: ```cs public class MyJobMiddleware(ILogger logger) : IJobMiddleware { public async Task ExecuteAsync(JobDescriptor jobDescriptor, Func> executeJob) { using var logScope = logger.BeginScope(new Dictionary {{ "JobName", jobDescriptor.Name }}); return await executeJob(); } } // Register the middleware builder.Services.AddScoped(); ``` ## Organizing the Admin UI If you have a lot of jobs you can group jobs in the Admin UI by a category, just like you can with queues. Set a category in the `Job` attribute like this: ```cs [Job("myjob", Category = "My category")] public class MyJob : IScheduledJob { } ``` Or set the category through the Admin UI under the More button on the job details page. ## Job loop delay Readiness events can wake the job loop immediately. Queue maintenance runs independently; see [queue maintenance](../queues/maintenance.md) for retry/lease timing and cleanup. By default Nexus will check once per second if there are any new jobs to start since it's not possible to schedule jobs to run more often than that. But in some situations you might want to lower that delay, eg if you use schedule extra runs frequently or if your queue jobs use `MaxNumberOfMessagesPerRun` and you want the job to restart as fast as possible. In such cases you can configure the job loop delay like this: ```cs builder.Services.AddNexus().AddJobs(options => { options.JobLoopDelay = TimeSpan.FromSeconds(0.5); }); ``` --- # Job logs Nexus has a [`ILoggerProvider`](https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.logging.iloggerprovider?view=net-8.0) which will collect logs by job run and store in the database. Nexus is compatible with any logging framework such as Serilog or NLog as long as it supports external providers. With Serilog writing to providers is disabled by default so you need to enable it to get Nexus logs to work: ```cs builder.Host.UseSerilog((context, services, configuration) => { configuration.ReadFrom.Configuration(context.Configuration); // This one is important }, writeToProviders: true); ``` ## `IJobLogWriter` Collected job logs are continuously flushed to the [`IJobLogWriter`](https://github.com/Commerce-Mind-Nexus/Nexus/blob/main/src/job-engine/Nexus.Abstractions/Jobs/IJobLogWriter.cs) by the job executor as they occur. There's three implementations of this interface built in. One for Postgres, one for SQL Server and one for SQLite and whichever database you've chosen to store job meta data is also used for logs. Logging to the database is enabled by default, but you can turn it off like this: ```cs builder.Services.AddNexus().AddJobs(options => { options.SaveLogsToDatabase = false; }); ``` Job logs are then stored in the database for as long as you'e set the retention for historical runs. When a historical run is deleted the logs for that run is also deleted. The default is to store historical runs and logs for 30 days, but you can configure it like this: ```cs builder.Services.AddNexus().AddJobs(options => { options.DefaultHistoricalRunsRetention = TimeSpan.FromDays(10); }); ``` You can also specify this per job with the `[Job]` attribute like this: ```cs [Job("MyJob", HistoricalRunsRetention = "30.00:00")] public class MyJob : IScheduledJob { ... } ``` The string you set is any string that can be passed to `TimeSpan.Parse()` or the special string `forever` to indicate that Nexus should never delete the job history. ## `IJobLogReader` When requesting job logs for a specific run the [`IJobLogReader`](https://github.com/Commerce-Mind-Nexus/Nexus/blob/main/src/job-engine/Nexus.Abstractions/Jobs/IJobLogReader.cs) service is called. The built-in implementations that store logs in the database also implement this interface. Calling `AddNexus().AddJobs()` registers both the writer and reader. If you send your logs to an external source like Elastic Search or Application Insights you can provide your own implementation of this interface. :::note[Note] You can view the job logs in the [admin UI](../admin-ui/index.md) where all logs for a single job run are displayed to make troubleshooting a job easier ::: ## Elasticsearch If you're sending logs to Elasticsearch using [`Elastic.Serilog.Sinks`](https://www.elastic.co/docs/reference/ecs/logging/dotnet/serilog-data-shipper) you can install the package `CommerceMind.Nexus.Elastic` and read job logs from Elastic instead. After you've installed if you add a call to `AddElasticJobLogs()` like this: ```cs builder.Services .AddNexus() // ... other Nexus initialization calls .AddJobs(options => { var isLocalDev = ...; options.SaveLogsToDatabase = isLocalDev; }) .AddElasticJobLogs(options => { options.ElasticUrl = "https://elastic:mypassword@localhost:9200"; options.LogIndexName = (serviceProvider) => "myindex"; // Or if you want complete control of the client settings: options.ClientSettings = (serviceProvider) => new ElasticsearchClientSettings(new Uri("https://elastic:mypassword@localhost:9200")) .DefaultIndex("myindex"); }) .Build(); ``` If you're sending logs to Elastic using some other transport than [`Elastic.Serilog.Sinks`](https://www.elastic.co/docs/reference/ecs/logging/dotnet/serilog-data-shipper) it might not work as expected because your transport might not be creating Elastic documents in the same way as the Serilog sink. If it doesn't work for you then check the property names that the job name, correlation id and log level is written to and set them in the configuration function like this: ```cs .AddElasticJobLogs(options => { options.CorrelationIdPropertyName = "..."; options.JobNamePropertyName = "..."; options.LogLevelPropertyName = "..."; }) ``` Please reach out to us and we'll try to modify the log reader to accommodate your transport as well. :::note[Note] CommerceMind.Nexus.Elastic currently only works with Elastic 8. If you're running Elastic 9 on the server you'll need to copy the implementation of the log reader using the Elastic 9 SDK as it's not possible for Nexus to support multiple major versions. ::: --- # Jobs with parameters In some situations it is useful for a job to take parameters: a data bag supplied from outside the job run. Some example use cases are: - A job that fetches data from an external system based on a change date can include a `DateTime` in its parameters and pass a date to the next execution. The future run can then fetch only changes that happened after the previous run. - Parameters can be supplied explicitly when starting a job from the Admin UI, allowing a manual run to affect how the job is executed. - An occasionally useful side effect, such as writing detailed troubleshooting output, can be controlled for an individual run instead of through application configuration. - A user can upload a file that the job reads during its execution. ## Defining a parameterized job Define a scheduled job that takes parameters by implementing [`IScheduledJob`](https://github.com/Commerce-Mind-Nexus/Nexus/blob/main/src/job-engine/Nexus.Abstractions/Jobs/IScheduledJob.cs). For example: ```cs public class MyJobParameters { public DateTime LastRunDate { get; set; } } public class MyJobWithParameters : IScheduledJob { public string DefaultSchedule => CronSchedule.TimesPerMinute(10); public MyJobParameters DefaultParameters => new(); public async Task ExecuteAsync(MyJobParameters parameters, CancellationToken cancellationToken) { var now = DateTime.UtcNow; var changes = await _service.GetChangesAsync(from: parameters.LastRunDate, to: now); // Do something interesting with the changes return JobResults.Completed($"Processed {changes.Count} changes", new MyJobParameters { LastRunDate = now, }); } } ``` The parameters class can be any class that can be serialized to and from JSON. Nexus stores default, next-run, retry, and manually supplied parameters as JSON. No separate database schema is created for a parameters class. To customize parameter serialization, register an implementation of [`IJobParametersSerializer`](https://github.com/Commerce-Mind-Nexus/Nexus/blob/main/src/job-engine/Nexus.Abstractions/Jobs/IJobParametersSerializer.cs) in the dependency injection container. ## Editing parameters in the Admin UI When Nexus can describe a job's parameter properties, the Admin UI offers two modes: - **Fields** is selected initially and renders controls appropriate for each property. - **JSON** provides direct access to the complete parameters object for technical users. The same editor is used when starting a job and when editing its default or next-run parameters on the job details page. The start-job editor is initialized from the saved default parameters, falling back to the job's parameter example when no defaults are available. Changes are preserved when switching modes. Switching from JSON to Fields requires a valid JSON object. Fields are validated before switching to JSON or submitting the form. File selection is available only in Fields mode. ### Supported fields Nexus describes public properties that have both a public getter and public setter. Read-only, indexed, and JSON-ignored properties are omitted. Objects, collections, and scalar types without a dedicated control are represented by an individual JSON editor rather than being recursively expanded. The JSON tab always shows the serialized representation and does not perform timezone conversion or provide file pickers. ### Labels, descriptions, and serialized names By default Nexus creates a label from the property name. Attributes can provide a better label, help text, or serialized property name: ```cs public class ImportParameters { [DisplayName("Changes from")] [Description("Only import changes made on or after this date.")] [JsonPropertyName("fromDate")] public DateOnly From { get; set; } } ``` `DisplayNameAttribute` and `DisplayAttribute` control the label, while `DescriptionAttribute` adds help text. `JsonPropertyNameAttribute`, Newtonsoft.Json's `JsonPropertyAttribute`, and the configured System.Text.Json property naming policy are respected when available. Both System.Text.Json and Newtonsoft.Json ignore attributes are respected. The API exposes this metadata in `JobSummary.ParameterFields`. `JobSummary.FileUploadsEnabled` tells clients whether file controls can currently accept uploads. ## Default parameters `DefaultParameters` should return an instance of the parameters class. It is passed to `ExecuteAsync()` when Nexus has no other parameters to use, such as for the first run or when the previous run did not provide parameters for the next run. Default parameters can be edited in either Fields or JSON mode on the job details page. Values saved in the Admin UI override the `DefaultParameters` property on the job class. To remove the saved override and use the value from code again, use the Reset action on the job details page. ## Next-run parameters A job can pass parameters to its next execution by returning them in its `JobResults`, as shown in the first example. This is optional. The next-run parameters can also be edited in Fields or JSON mode on the job details page. If neither a previous run nor a manual start supplies parameters, Nexus uses the default parameters. ## Parameters and job failure If a non-manual job run with parameters fails, its next run receives the same parameters. In the first example, `LastRunDate` therefore contains the date set by the last successful run. To use different parameters after a failure, catch the error and pass explicit parameters to `JobResults.Failed()`: ```cs public async Task ExecuteAsync(MyJobParameters parameters, CancellationToken cancellationToken) { var now = DateTime.UtcNow; try { // Implementation excluded for brevity } catch (Exception e) { return JobResults.Failed(e, new MyJobParameters { LastRunDate = now, }); } } ``` The run is still marked as failed, but the next run receives the parameters supplied to `JobResults.Failed()`. ## Uploaded files Use [`JobFile`](https://github.com/Commerce-Mind-Nexus/Nexus/blob/main/src/job-engine/Nexus.Abstractions/Jobs/JobFile.cs) for a file parameter: ```cs public class ImportParameters { [Description("The CSV file to import.")] public JobFile InputFile { get; set; } = null!; } public class ImportJob : IScheduledJob { public string DefaultSchedule => CronSchedule.NotScheduled(); public ImportParameters DefaultParameters => new(); public async Task ExecuteAsync(ImportParameters parameters, CancellationToken cancellationToken) { var contents = parameters.InputFile.ReadAsText(); // Import the contents return JobResults.Completed(); } } ``` ### Configuring filesystem storage File uploads are disabled by default. Set `UploadedFilesPath` when registering scheduled jobs to enable the built-in filesystem storage: ```cs builder.Services .AddNexus() .AddJobs(options => { options.UploadedFilesPath = @"\\fileserver\shared\nexus-job-files"; options.UploadedFileRetention = TimeSpan.FromDays(14); options.UploadedFileCleanupInterval = TimeSpan.FromHours(1); }); ``` Use a shared folder when a job can be uploaded on one application instance and executed on another. The built-in storage uses generated opaque filenames and never uses the uploaded filename as a filesystem path. Storage keys are validated before paths are resolved. `UploadedFileRetention` defaults to seven days, and `UploadedFileCleanupInterval` defaults to one hour. A hosted cleanup loop sweeps once at startup and then at the configured interval. It also removes uploads abandoned when a user selects a file but never successfully starts or saves the job parameters. Uploaded references can be reused by retries and later runs only until cleanup removes the underlying file. Configure retention to be longer than the maximum expected delay before a run, including retries. Be especially careful when placing uploaded references in default or next-run parameters, because those references do not make the file permanent. When uploads are disabled, the Admin UI keeps file controls disabled and explains that `UploadedFilesPath` or a custom storage implementation is required. ### Using custom file storage Implement [`IJobFileStorage`](https://github.com/Commerce-Mind-Nexus/Nexus/blob/main/src/job-engine/Nexus.Abstractions/Jobs/IJobFileStorage.cs) to store files somewhere else, such as Azure Blob Storage: - `IsEnabled` reports whether the storage is available for uploads. - `UploadAsync()` stores the supplied stream and returns a `JobFile` containing a persistable opaque reference. - `OpenReadAsync()` reconstructs a readable stream from that reference before job execution. - `DeleteOlderThanAsync()` deletes storage objects older than the supplied UTC cutoff. File storage is resolved in a scope and may depend on scoped application services. Register the custom implementation before adding jobs so it replaces the default registration: ```cs builder.Services.AddScoped(); builder.Services .AddNexus() .AddJobs(options => { options.UploadedFileRetention = TimeSpan.FromDays(30); }); ``` `UploadedFilesPath` is not required for custom storage. The same retention and cleanup interval options control calls to the custom implementation's `DeleteOlderThanAsync()` method. --- # Job progress Some jobs are expected to take a long time to complete and for such jobs it's nice to be able to see the progress for the job. Nexus has built-in support for jobs to report their progress through the `IJobProgress` interface like this: ```cs public class ExampleJob : IScheduledJob { private readonly IJobProgress _jobProgress; public ExampleJob(IJobProgress jobProgress) { _jobProgress = jobProgress; } public async Task ExecuteAsync(CancellationToken cancellationToken) { var thingsToProcess = await GetThingsToProcessAsync(); _jobProgress.SetTotal(thingsToProcess.Count); foreach (var thingToProcess in thingsToProcess) { cancellationToken.ThrowIfCancellationRequested(); await ProcessAsync(thingToProcess); _jobProgress.Increment(1); } return JobResults.Completed(); } } ``` The important lines here are `IJobProgress.SetTotal(thingsToProcess.Count);` and `IJobProgress.Increment(1);`. The call to `IJobProgress.SetTotal()` tells Nexus how much work the job expects to do and `IJobProgress.Increment(1);` tells Nexus that we've done some work. The numbers to use here are up to you. If your job does multiple things and some operations take a longer time to complete than others you can increment the progress with a higher number than `1`. You can call the `SetTotal()` method multiple times during a job. Eg if the job sees that there's more work to do than first expected. If you increment the progress more than what the total has been set to the total will automatically increase. ## Queue job progress Queue jobs do not report progress automatically by default, because calculating a total adds a queue count before processing can begin. Implement `IQueueJobBase.AutomaticProgressMode` to opt into count-only or percentage progress. See [automatic queue job progress](../queues/jobs.md#automatic-progress-reporting) for the modes and their performance tradeoffs. This setting only controls progress reported by the queue executor. A queue job can inject and use `IJobProgress` directly regardless of the selected mode. ## Progress message The progress can report an optional message as well. Something like this: ```cs public class ExampleJob : IScheduledJob { private readonly IJobProgress _jobProgress; public ExampleJob(IJobProgress jobProgress) { _jobProgress = jobProgress; } public async Task ExecuteAsync(CancellationToken cancellationToken) { _jobProgress.SetMessage("Calculating work to be done..."); var thingsToProcess = await GetThingsToProcessAsync(); var otherThingsToProcess = await GetOtherThingsToProcessAsync(); _jobProgress.SetTotal(thingsToProcess.Count + (otherThingsToProcess.Count * 2)); _jobProgress.SetMessage("Processing things..."); foreach (var thingToProcess in thingsToProcess) { cancellationToken.ThrowIfCancellationRequested(); await ProcessAsync(thingToProcess); _jobProgress.Increment(1); } _jobProgress.SetMessage("Processing other things..."); foreach (var otherThingToProcess in otherThingsToProcess) { cancellationToken.ThrowIfCancellationRequested(); await ProcessAsync(otherThingToProcess); _jobProgress.Increment(2); } return JobResults.Completed(); } } ``` Before we know how much work we have to do, we set a progress message which at least feedbacks that's something happening and where we are in the job execution. In the above example we also give `otherThingsToProcess` a weight of `2` instead of `1` because we anticipate that they will take a longer time to process. Which gives a better indication as to when the job will be finished. For some jobs you might not have numbers to increment but you still want to give feedback to what's happening. In such cases you can use just the progress message to indicate what the job is doing. The admin UI will display the message but won't show a progress bar. Note that the message isn't cleared if you increment progress or change the total. If you wish to clear the current message you'll need to call `IJobProgress.SetMessage(null)`. ## Calling `IJobProgress` outside of a job Most of the time you will call `IJobProgress` directly from inside the job implementation but there's nothing stopping you from calling it inside other services. If you call `IJobProgress` from a service when the service isn't invoked in a job context it will simply not doing anything. `IJobProgress` stores a [`AsyncLocal`](https://learn.microsoft.com/en-us/dotnet/api/system.threading.asynclocal-1?view=net-7.0) instance connected to it's current running job and if there's no context the call to `IJobProgress` will simply not do anything. If you try to get a hold of `IJobProgress` in an assembly or application which hasn't registered the jobs system with `IServiceCollection.AddNexus().AddJobs()` you'll need to inject your own mock implementation of the interface as otherwise you'll get an exception that there's no registered implementation of the interface. --- # Functions Nexus functions are a lightweight way of executing a method on a service in the background. Sometimes you want to execute some method but you want it to run in the background. Either because the method takes a long time or because you want to make sure that it's retried until it completes successfully. Let's take a simplified example: ```cs public class OrderController : Controller { private readonly IOrderConfirmationEmailService _orderConfirmationEmailService; public OrderController(IOrderConfirmationEmailService orderConfirmationEmailService) { _orderConfirmationEmailService = orderConfirmationEmailService; } public async Task OrderPlaced(string orderId) { await _orderConfirmationEmailService.SendEmailForOrderAsync(orderId); return Json(new { Success = true, }); } } ``` We need to send the order confirmation email, but there's a couple of issues with this approach. What happens if `SendEmailForOrderAsync()` throws an error? And what if it takes a couple of seconds to complete? If it throws an exception we'll miss sending the email for that order. And if it takes a couple of seconds we'll slow down the experience for the end customer. We can fix that by using a Nexus function instead, like this: ```cs public class OrderController : Controller { private readonly INexusFunction _nexusFunction; public OrderController(INexusFunction nexusFunction) { _nexusFunction = nexusFunction; } public async Task OrderPlaced(string orderId) { await _nexusFunction.RunInBackgroundAsync(x => x.SendEmailForOrderAsync(orderId)); return Json(new { Success = true, }); } } ``` With this change we've moved the actual execution of the `SendEmailForOrderAsync()` method into the background (maybe even to a different server). When Nexus executes the method it will catch any exceptions that happen and retry the call later. You can control the retry behavior using the `[NexusFunction]` attribute which you can read about below. ## Registering functions The only things you need to do is to call `AddFunctions()` like this: ```cs builder.Services.AddNexus().AddFunctions(); ``` And then install either `CommerceMind.Nexus.Sqlite`, `CommerceMind.Nexus.Postgres`, or `CommerceMind.Nexus.SqlServer` and initialize it: ```cs builder.Services .AddNexus() .AddSingleServerInstanceEvents() // Use a shared broker for multiple instances .AddFunctions() // Either: .AddSqliteConnection() // Or: .AddPostgresConnection() // Or: .AddSqlServerConnection() .Build(); ``` After that you can use `INexusFunction` and start running methods in the background. Note that by default the application will also start to run functions in the background. If you want an application to only add functions but not run them you can set `UseBackgroundService` to false like this: ```cs builder.Services.AddNexus().AddFunctions(options => { options.UseBackgroundService = false; }); ``` You must have at least one application that runs functions so this is only if you have multiple applications where one or more is dedicated for background processing. If you're not using a continously running server and instead using something like Azure Functions or Azure Container Instances that start on a schedule you can set `UseBackgroundService` to `false` and instead call `INexusFunctionExecutorService.RunAllFunctionsPendingSinceLastCallAsync()` from your entrypoint. That method will run all functions that have been scheduled to run since the last time that method was called. See the example [Azure function trigger here](../faas.md). :::note[Keeping processed functions] By default Nexus will store meta data about processed functions for one day in the database before deleting them. You can control this by setting `AddNexus().AddFunctions(options => options.KeepProcessedRunsFor = TimeSpan.Zero);`. You can also set the retention per function using the `[NexusFunction]` attribute which you can read about below. ::: ## Service registration You can use any service you want when calling `RunInBackgroundAsync()`. The only requirement is that the service is registered in the `IServiceCollection`. Which means that you can use interfaces as well as long as you register an implementation for it. So for the `IOrderConfirmationEmailService` example above you'd do something like this: ```cs serviceCollection.AddScoped(); ``` :::note[Multi threading] Each function execution resolves its target service in a fresh dependency injection scope. Scoped services can inject scoped Nexus services and a DbContext. Different function executions can run concurrently, so shared singleton dependencies must remain thread safe. See [dependency injection](../general/dependency-injection.md) for calling Nexus from singleton application services. ::: ## Invoking functions in the Admin UI If you want to be able to invoke functions yourself through the Admin UI you need to make Nexus aware of which methods you intend to use as a function. Nexus allows invoking any method using `INexusFunction` in code, but requires your methods to be decorated with the `[NexusFunction]` attribute in order to list them as a function in the admin UI. When invoking a function in the admin UI you input the arguments to the method as JSON. ## Method arguments Nexus will automatically capture the values of arguments passed to the method like in the example above with the `orderId` argument. The only requirement is that it can only be values that can be serialized to JSON. Which means that you can pass strings, numbers, dates, and data objects to the service method. But you can't pass another service for example. If you have a method that requires another service as an argument you need to create a separate Nexus friendly method. Something like this: ```cs public class ExampleService { private readonly OtherService _service; public ExampleService(OtherService service) { _service = service; } public void ExampleMethod(OtherService service, string someString) { service.SomeMetod(someString); } public void ExampleMethod(string someString) { ExampleMethod(_service, someString); } } ``` So instead of having Nexus call the `ExampleMethod(OtherService service, string someString)` method you'd let it call `ExampleMethod(string someString)`. You can of course create a separate service that does this instead of having the method in the same service. If you need to change the way arguments are serialized you can register a custom implementation of `INexusFunctionArgumentsSerializer` with the service provider. ## Cancellation token The methods on `INexusFunction` has overloads that passes a cancellation token to the provided function. This lets you pass the cancellation token on to your service method like this: ```cs await _nexusFunction.RunInBackgroundAsync((s, ct) => s.LongOperation(ct)); ``` The cancellation token passed to the service is the [graceful shutdown](../general/hosting.md#graceful-nexus-shutdown) token that will be triggered either if a graceful shutdown has been initiated or if the application is shutting down because of a deploy or server restart. ## Running a method after a delay If you don't want to run the method as fast as possible you can call `RunInBackgroundAfterAsync()` instead of `RunInBackgroundAsync()`. Like this: ```cs await _nexusFunction.RunInBackgroundAfterAsync(TimeSpan.FromMinutes(10), x => x.SendEmailForOrderAsync(orderId)); ``` ## Running a method until a condition is met Sometimes you want to keep running a method until some condition is true such as polling an external system. Let's say that you don't want to send the order confirmation email until the customer on the order has been exported to the CRM. You can use recursion to let a method called by Nexus schedule a new run, something like this: ```cs public class MyOrderConfirmationEmailService : IOrderConfirmationEmailService { private readonly ICustomerService _customerService; private readonly IOrderService _orderService; private readonly INexusFunction _nexusFunction; public MyOrderConfirmationEmailService(ICustomerService customerService, IOrderService orderService, INexusFunction nexusFunction) { _customerService = customerService; _orderService = orderService; _nexusFunction = nexusFunction; } public async Task SendEmailForOrderAsync(string orderId) { var order = await _orderService.GetAsync(orderId); if (!await _customerService.ExistsAsync(order.CustomerNumber)) { // Wait 5 minutes to see if the customer has been exported await _nexusFunction.RunInBackgroundAfterAsync(TimeSpan.FromMinutes(5), x => x.SendEmailForOrderAsync(orderId)); } else { // Customer exists, let's break the recursion and send the email await SendEmailAsync(order); } } private async Task SendEmailAsync(Order order) { // Send the email } } ``` In this example Nexus will continuously call `SendEmailForOrderAsync(orderId)` every five minutes until the customer has been exported. ## [NexusFunction] attribute The `[NexusFunction]` attribute lets you control some aspects of how Nexus will run a method. You specify the attribute either on the service class/interface or on individual methods. You can use `[NexusFunction]` to control for how long meta data about successful runs of a method is stored like this: ```cs [NexusFunction(KeepProcessedRunFor = "10")] public void ExampleMethod() { } ``` This will store all calls to this method for 10 days as the value for `KeepProcessedRunFor` must be a string that can be parsed to a `TimeSpan` using `TimeSpan.Parse()`. ### Retries By default Nexus will retry a function if an exception is thrown. The default strategy is to use linear back-off where the first retry is after 5 seconds, the second retry is after 10 seconds, the third is fter 15 seconds, etc. You can control the delay and the max number of retries globally by setting the `RetryAfter` and `MaxRetries` options like this: ```cs builder.Services.AddNexus().AddFunctions(options => { options.RetryAfter = TimeSpan.FromSeconds(10); options.MaxRetries = 5; }); ``` But you can also set the max retry count on a function level like this: ```cs [NexusFunction(MaxRetries = 5)] public void ExampleMethod() { } ``` By default Nexus will keep calling the method forever until it completes successfully without any max retry number. You can change the retry strategy from linear back-off to either `NexusFunctionRetries.RetryUntilSuccess` or `NexusFunctionRetries.ManualRetry`. `NexusFunctionRetries.RetryUntilSuccess` will keep calling the method without any delay whereas `NexusFunctionRetries.ManualRetry` won't retry at all. You can trigger a retry manually through the Admin UI or the API. ```cs [NexusFunction(Retries = NexusFunctionRetries.RetryUntilSuccess)] public void ExampleMethod() { } ``` ## Function middlewares In some cases you want to wrap the execution of a function. You might want to add additional log context properties or instrument the function execution in some way. To achieve this you can register one or more `INexusFunctionMiddleware` instances in the service collection. Eg: ```cs public class MyFunctionMiddleware(ILogger logger) : INexusFunctionMiddleware { public async Task ExecuteAsync(NexusFunctionDescriptor nexusFunctionDescriptor, Func> executeFunction) { using var logScope = logger.BeginScope(new Dictionary {{ "FunctionName", nexusFunctionDescriptor.DisplayName }}); return await executeFunction(); } } // Register the middleware builder.Services.AddScoped(); ``` ## How Nexus Functions works When you call `RunInBackgroundAsync(x => x.SomeMethod(someArgument))` Nexus will use .NET reflection to gather information about what it is you want to run. The full name of the service type, the name of the method to use and the arguments passed. This is then stored in the database in a row that looks something like this: | status | service_type | method | parameter_types_json | arguments_json | run_at_utc | | --------- | --------------------------------------------------------- | ------------------------ | ------------------------- | -------------- | --------------------- | | "Pending" | "My.Namespace.IOrderConfirmationEmailService, MyAssembly" | "SendEmailForOrderAsync" | ["System.String, System"] | ["123"] | "2022-01-01T00:00:00" | Nexus will then read this data from the database and again use reflection to locate the .NET types needed and deserialize the arguments from JSON. It will then resolve the service from a fresh dependency injection scope and execute the method in a [background service](https://learn.microsoft.com/en-us/aspnet/core/fundamentals/host/hosted-services?view=aspnetcore-6.0&tabs=visual-studio#backgroundservice-base-class). The type signature for `RunInBackgroundAsync()` is a lambda that gets the `TService` passed in and either returns a `Task` or nothing. So the type system allows you to write something like this: ```cs await _nexusFunction.RunInBackgroundAsync(x => orderId == "123" ? x.SendEmailForOrderAsync(orderId) : x.DoSomethingElseAsync(orderId)); ``` Nexus will however throw an exception if you do this as the only allowed lambda to pass in is a method call on the service passed into the lambda. You can think of it as a more readable and type safe way of writing this: ```cs await _nexusFunction.RunInBackgroundAsync(typeof(IOrderConfirmationEmailService), nameof(IOrderConfirmationEmailService.SendEmailForOrderAsync), new object?[] { orderId }); ``` ### Return value The method you use in your lambda is allowed to return a value for synchronous methods and `Task` or `Task` for asynchronous methods but Nexus won't do anything with the return value. If you need to do something with the return value you should wrap the method in another method that does something with it. If you want to call a method that returns `false` if it didn't go as expected you should wrap it in a method that throws an exception instead. Something like this: ```cs public class ExampleService { public bool TryDoSomething() { // Do something interesting here and return true/false } [NexusFunction] public void DoSomething() { if (!TryDoSomething()) { throw new Exception("It didn't go as planned"); } } } ``` ## Renaming classes or methods Since Nexus will store full type names such as `My.Namespace.MyService, MyAssembly` in the database it's important not to rename classes that are used as Nexus functions. If you rename such a class or method existing functions will fail and you need to update the database with the new type names. --- # Functions vs Jobs & Queues If you've already read about [jobs](../jobs/index.md) and [queues](../queues/index.md) you might ask yourself "when should I use functions and when should I use jobs and queues?". There's a lot of similarities between them and both can often be used for the same use case. You can think of functions as a lightweight version of jobs & queues that are well suited for use cases where creating a job and a queue feels like overkill. A key difference is that functions can only be created by .NET code and you can't have an external system adding a function call through the API like you can with [queues](../queues/enqueueing.md). Another big difference is that queues allows you to separate concerns in a better way. The part of the application that creates a queue message has no idea what will happen when that message is processed, but when you use a Nexus function the caller has to specify what service should get called. There's also no way of doing batching with functions like you can with queue jobs. That is, if 100 calls to the same method gets scheduled there's no way of grouping them into a single call to the service. Queues also offer a choice of execution model: [queue jobs](../queues/jobs.md) provide one active job run, schedules, parameters, progress, and mutexes; [distributed processors](../queues/processors.md) scale independent work across worker pools on every instance. [Named queues](../queues/named.md) separate destinations while sharing one message contract. A rule of thumb is to start with a Nexus function if that works for your use case, and later on move it to a queue job if you need to. :::note[Tip: Use only functions] The functions, jobs and queues sub systems are independant from each other and can be used in isolation. So if you don't have any need for jobs & queues you can skip registering those sub systems and only call `builder.Services.AddNexus().AddFunctions()`. ::: --- # Queue activation Queue activation records that a queue may contain work ready to process. It lets a queue job start after enqueueing without a cron schedule or a separate start request. Repeated requests are coalesced per queue, so a burst of messages does not create a separate job run for every enqueue. ## Choosing how a queue runs | Processing model | When processing starts | Concurrency | | --- | --- | --- | | Queue job with no schedule | When messages become ready | One active job run across instances, with optional processing threads | | Queue job with a cron schedule | On the saved schedule, or an explicit manual/extra run | One active job run across instances | | Distributed queue processor | When work is ready, with periodic reconciliation | A worker pool on every instance | An ordinary queue job needs no `DefaultSchedule`: ```cs public class OrderExportJob : IQueueJob { public async Task ProcessMessageAsync( OrderExportMessage message, CancellationToken cancellationToken) { await ExportOrderAsync(message, cancellationToken); return ProcessResults.Processed(); } } // Enqueueing is enough; there is no separate enqueue-and-start method. await enqueuer.EnqueueAsync(new OrderExportMessage { OrderId = "123" }); ``` Add a cron `DefaultSchedule` when the queue should run only at those times. Schedules are saved in the database when the job is first registered. To change an existing job to readiness-driven processing, clear its saved schedule through the Admin UI/API or `IJobMetaDataRepository.UpdateJobScheduleAsync(jobName, null)`. Removing the property from the class alone does not clear that saved schedule. An empty schedule means readiness-driven processing for queue jobs. For an ordinary `IScheduledJob`, an empty schedule still means it runs only when explicitly started. ## What happens after enqueueing? For a ready message, the normal enqueuer requests activation after the queue operation completes. With the [Entity Framework integration](entity-framework.md), this happens after the context's transaction commits. Rolled-back operations do not activate the queue, and in-memory additions finish before consumers are notified. Nexus records activation state through `IQueueActivationStore` and sends a best-effort instance event to wake processing quickly. Losing that event does not lose a persisted activation. A pending activation can wait while the queue job is disabled or paused on error; processing resumes when it is eligible again. The activation is a signal to inspect the queue, not a count or snapshot of messages. It may result in no work if another consumer already handled the messages. Messages arriving during processing retain readiness for more work; they do not start a concurrent run of the same job. Activation is separate from message leasing. [Leases](index.md#leasing-messages) still protect messages while they are being processed, and expired leases can cause redelivery. Processing handlers must tolerate the retry behavior of their storage provider. ## Activation delay versus processing delay Use `ActivationDelay` to collect a short burst before starting a readiness-driven queue: ```cs [Queue("order_exports", ActivationDelay = "250ms")] public class OrderExportMessage : IQueueMessage { public string OrderId { get; set; } = ""; } ``` The first request starts a fixed collection window. Additional messages do not keep extending it. It does not delay each message and does not replace a cron schedule. `ProcessingDelay`, `EnqueueContext.ProcessAfter`, and `ProcessAtUtc` instead determine when an individual message becomes eligible. [Queue maintenance](maintenance.md) promotes due messages and requests readiness; a scheduled-only job still waits for its schedule. Named/generated queue definitions also support `ActivationDelay`. A saved override can be changed through `IQueueMetaDataRepository.SetActivationDelayAsync()` or `POST /api/queues/{queueName}/activation-delay` with an `activationDelay` string; a null value restores the code default. ## Activation storage Activation storage and message storage are separate choices. SQL Server, PostgreSQL, and SQLite provider registration supplies database activation storage automatically. No additional activation registration is needed for the usual database-backed application. Redis can hold activation state instead: ```cs builder.Services.AddNexus().AddRedisQueueActivations(options => { options.ConnectionString = redisConnectionString; options.KeyPrefix = "my-nexus-app"; }); ``` This is independent of Redis external pending storage and Redis instance events. Configure the same shared activation store and namespace on producers and processing hosts for the same queues. Redis persistence and availability also determine how well its stored activation survives infrastructure failures. For a single-instance application, `AddMemoryQueueActivations()` opts into process-local activation state. It does not survive restart or coordinate separate producer/worker processes. It is distinct from `QueueStorageMode.InMemory`, which selects where messages are processed. Only one explicitly selected activation provider can be registered. ## Recovery and external producers Queue writes and activation requests are distinct steps; even database-backed activation does not imply an atomic enqueue-plus-activation transaction. The queue-job reconciler checks readiness-driven queues on startup and periodically to recover missed activation. Configure its interval with `JobsOptions.QueueActivationReconciliationInterval` (default ten minutes). Unhandled queue-job failure/cancellation defers activation using `QueueActivationFailureDelay` (default five seconds). Individual message retries still use their own retry dates. Distributed processors perform their own reconciliation through `QueueProcessorOptions.ReconciliationInterval`. Delayed-message and lease maintenance has separate timing; see [queue maintenance](maintenance.md). External pending providers have [durable readiness bridges](external-pending.md#readiness-bridges-and-direct-producers). A direct broker producer must use that provider's current topology/envelope; writing only a processing copy is not enough for prompt readiness-driven processing. Database migration does not transfer messages from old broker resources. For custom infrastructure operations or explicit database transactions, `IQueueReadinessNotifier.NotifyAsync(queueName)` records readiness. Call it after a successful commit when the normal integration does not own the commit. It does not enqueue a message or wait for processing to finish. Normal application enqueueing should use the enqueuer's built-in notification. `IJobStartRequester` is a separate best-effort job trigger. It does not provide the persistent queue-readiness contract. See [starting a job programmatically](../jobs/index.md#starting-a-job-programatically) for durable extra runs. --- # Processing directly from Azure Service Bus topics Nexus allows you to use Azure Service Bus as an external pending storage, which means that Nexus reads and processes messages directly from Service Bus without first storing them in an SQL-backed Nexus queue. Use it by first installing `CommerceMind.Nexus.Azure` and then configuring the external storage: ```cs builder.Services .AddNexus() .AddAzureServiceBusExternalPendingStorage(options => { options.ConnectionString = builder.Configuration.GetConnectionString("azureServiceBus"); }); ``` And then on your queue: ```cs [Queue("export_order", StorageMode = QueueStorageMode.ExternalPending)] public class ExportOrderQueueMessage : IQueueMessageWithId { } ``` For each Nexus queue, v3 uses a topic with two durable subscriptions. With the defaults, queue `export_order` maps to topic `nexus-export_order`, processing subscription `processing`, and readiness subscription `readiness`. Nexus leases and settles messages only through the processing subscription. The readiness bridge completes its copy only after activation is durable. Direct producers must publish to the topic. A legacy Service Bus queue cannot provide passive durable readiness observation, so migrate publishers to the topic before changing the Nexus queue job to readiness-only mode. Creating the topic and subscriptions requires Manage permission when Nexus creates topology. Runtime send/listen permissions can be narrower after provisioning. Read more about [external pending storage here](./external-pending.md#readiness-bridges-and-direct-producers). # Enqueueing from Azure Service Bus queues A common scenario is to have queues in Azure Service Bus that other systems sends messages to. Since many systems already have integrations to Azure Service Bus it makes it easy to consume events from such systems. An option is of course to use the `Azure.Messaging.ServiceBus` NuGet package and process the messages using a `ServiceBusProcessor` and add any logic you need into the `ProcessMessageAsync` event handler to handle the message. Another option is to use the `CommerceMind.Nexus.Azure` package to take the messages from the service bus queue and place them in a Nexus queue. Doing so lets you use [virtual queues](./virtual.md), [idempotent messages](./index.md#idempotent-messages) and [to store processed messages](./retention.md) which can be very useful. But even if you don't need any of those features you'll benefit from the visibility, introspection and error handling of Nexus queues. ## Reading from an Azure queue or topic into a Nexus queue Start by installing the `CommerceMind.Nexus.Azure` NuGet package. After that you configure it like this in your `Program.cs`: ```cs builder.Services .AddNexus() .AddQueues() .AddNexusAzureServiceBusQueues() .EnqueueFromAzureQueue("azure-queue") .Build() ; ``` By default Nexus will look for the connection string using `IConfiguration.GetConnectionString("azureServiceBus")` but you can add a loader func on the options object if you need to fetch it from somewhere else: ```cs builder.Services .AddNexus() .AddQueues() .AddNexusAzureServiceBusQueues(options => { options.ConnectionStringLoader = (serviceProvider, queueOrTopicName) => "Endpoint=sb://xxx.servicebus.windows.net/;..."; // Or: options.ServiceBusClientLoader = (serviceProvider, queueOrTopicName) => new ServiceBusClient(...); }) .EnqueueFromAzureQueue("azure-queue") .Build() ; ``` If you want to read from a Service Bus Topic you also need to add a subscription name loader like this: ```cs builder.Services .AddNexus() .AddQueues() .AddNexusAzureServiceBusQueues(options => { options.SubscriptionNameLoader = (serviceProvider, topicName) => "my-subscription"; }) .EnqueueFromAzureTopic("azure-topic") .Build() ; ``` With the above example you also need a `ExampleQueueMessage` defined that implements `IQueueMessage` or `IQueueMessageWithId`. Read more about that in [the Nexus queues intro](./index.md). The assumption is that the JSON on the Azure queue can be deserialized directly to the `ExampleQueueMessage` class. If that doesn't work for you it's possible to specify a separate type for the Azure message and also pass a function that maps the Azure message to the Nexus message like this: ```cs public class ExampleQueueMessage : IQueueMessage { public string SomeProperty { get; set;} } public class AzureQueueMessage { public string SomeOtherProperty { get; set;} } builder.Services .AddNexus() .AddQueues() .AddNexusAzureServiceBusQueues(options => { // This allows you to control things like which status or when messages are processed options.EnqueueContextFactory = message => new EnqueueContext { Priority = message is MyImportantQueueMessage ? 10 : 0 } }) .EnqueueFromAzureQueue( "azure-queue", m => new ExampleQueueMessage { SomeProperty = m.SomeOtherProperty } ) .Build() ; ``` The `CommerceMind.Nexus.Azure` package only expects there to be registered `IEnqueuer`s for the message types used which means that it can enqueue to any of the underlying enqueuers; [`IVirtualEnqueuer`](./virtual.md), [`IDatabaseEnqueuer`](./enqueueing.md#enqueueing-using-the-queueing-package), [a memory enqueuer](./in-memory.md) or [`IApiEnqueuer`](./enqueueing.md#enqueueing-using-the-apiclient-package). --- # Enqueueing from outside the processing host A website or API can produce messages while a separate Nexus application processes them. Share the message contract and stable queue names between the applications. There are three ways to publish. ## Enqueueing using the Queueing package Queueing is included in `CommerceMind.Nexus`; there is no separate v3 `CommerceMind.Nexus.Queueing` package. Shared message contracts can reference `CommerceMind.Nexus.Abstractions`: ```cs [Queue("order_exports")] public class OrderExportMessage : IQueueMessage { public string OrderId { get; set; } = ""; } ``` The processing host registers queues and their processing implementation: ```cs builder.Services.AddNexus() .AddDatabasePollingInstanceEvents() .AddQueues(options => { options.AssembliesToScanForQueueMessages.Add(typeof(OrderExportMessage).Assembly); }) .AddJobs(options => { options.AssembliesToScanForJobs.Add(typeof(OrderExportJob).Assembly); }) .AddPostgresConnection() .Build(); ``` `OrderExportJob` implements `IQueueJob` or `IBatchQueueJob`. Alternatively, register `AddQueueProcessors()` for a distributed processor instead of the queue job. The producing host registers the same queues and database, without registering processing: ```cs builder.Services.AddNexus() .AddDatabasePollingInstanceEvents(options => { options.BrokerType = InstanceEventBrokerType.ProduceEvents; }) .AddQueues(options => { options.AssembliesToScanForQueueMessages.Add(typeof(OrderExportMessage).Assembly); }) .AddPostgresConnection() .Build(); ``` Inject the scoped `IEnqueuer` into the producer's controller/application operation and call `EnqueueAsync()`. `AddQueues()` alone does not register jobs or distributed processors, though it does initialize queue schemas and run queue maintenance. Producers need migration permissions when initialization changes schema. Both hosts use the same database and [activation store](activation.md#activation-storage). Configure a shared [instance-event broker](../general/instance-events.md) to wake workers promptly; a producer-only broker avoids listening for those events in the producer. If using Redis activation storage instead of the database default, configure it consistently on both hosts. The queue job's saved schedule determines whether it starts on readiness or waits for its cron schedule. The producer does not need a separate enqueue-and-start call. With [EF integration](entity-framework.md), publishing to a database queue joins the producer's scoped unit of work. This approach requires the producer to access the Nexus database. Use the HTTP client when that access is undesirable. In-memory queues must be published through the API of their single owning instance, not by writing their fallback database from another application. ## Enqueueing using the ApiClient package `CommerceMind.Nexus.ApiClient` publishes through the Nexus HTTP API without local database access, queue storage, processing, or an instance-event broker. Configure the HTTP client and contract assemblies: ```cs builder.Services.AddHttpClient(ApiEnqueuerOptions.DefaultHttpClientName) .ConfigureHttpClient(client => { client.BaseAddress = new Uri("https://nexus-host.example.com/nexus/"); // Configure authentication headers/handlers here if needed. }); builder.Services.AddNexus().AddApiEnqueuers(options => { options.AssembliesToScanForQueueMessages.Add(typeof(OrderExportMessage).Assembly); }).Build(); ``` The base address must match the host's API base path. Ordinary queues can be reached through `IEnqueuer`; use `IApiEnqueuer` when explicitly selecting a remote named destination: ```cs await apiEnqueuer.EnqueueAsync("order_exports", message); ``` Named destinations do not need per-name client registrations. The server validates the destination and runs any virtual transformers on that host. A remote enqueue is immediate and cannot roll back with a producer's EF transaction; use a local database outbox when delivery must follow that transaction. Change `ApiEnqueuerOptions.HttpClientName` to use a different named HTTP client. To distinguish implementations, use `IApiEnqueuer`, `IVirtualEnqueuer`, or `IDatabaseEnqueuer` and their typed equivalents where available. These are still scoped services; singleton callers should [create an operation scope](../general/dependency-injection.md#calling-nexus-from-a-singleton-or-hosted-service). ## Enqueueing over HTTP using the API Non-.NET clients can call `POST /api/queues/{queueName}` or `POST /api/queues/{queueName}/batch`, relative to the configured API base path. The create-item body wraps the payload in `message`: ```json { "message": { "OrderId": "123" }, "priority": 10 } ``` Queue creation payloads no longer have `startProcessing`; readiness follows the server's processing mode. See the [API reference](/api/reference/nexus-api/) for batch and context fields. [Queue webhooks](webhooks.md) accept raw message JSON instead of the create-item wrapper. For named/generated destinations, always select the queue explicitly. To run generated `ShouldEnqueue` routing, send to its configured virtual `RoutingQueueName` rather than a concrete destination. See [named queues](named.md) and [generated queues](generated.md). --- # Entity Framework with Nexus queues If you're using Entity Framework you might want to sync your `SaveChangesAsync()` calls with calls to `EnqueueAsync()` on Nexus queues to ensure that they are committed together. To achieve this you can install the `CommerceMind.Nexus.EntityFramework` package and initialize it like this in Program.cs: ```cs builder.Services .AddNexus() .AddEntityFramework(); builder.Services.AddDbContext((serviceProvider, options) => { options.UseNexus(serviceProvider); }); ``` This will install interceptors with Entity Framework that instead of sending a Nexus message directly to the database will keep it in the background until you call `context.SaveChangesAsync()` and will then use the same database connection and transaction that EF uses for your other data. Note that if there's any messages to save Nexus will create an explicit transaction if there isn't already one. Which means that your EF data will be saved in the same transaction as your Nexus messages. The `DbContext` and Nexus enqueuer must be resolved from the same dependency injection scope. Nexus resolves the configured context from that scope when a message is enqueued, so each request or unit of work keeps its own context and pending messages. The context must be registered as scoped, must not use DbContext pooling, and must call `UseNexus()` with the same scoped service provider. Use `SaveChangesAsync()` and asynchronous transaction commits while queue operations are pending. Nexus rejects their synchronous counterparts because queue persistence is asynchronous. Messages that are enqueued but never saved with `SaveChangesAsync()` (or an asynchronous commit) are discarded together with the context, just like unsaved entity changes. If you need to enqueue without saving the context, see [Enqueueing outside an Entity Framework unit of work](#enqueueing-outside-an-entity-framework-unit-of-work). Readiness is requested once per ready queue after a successful commit, even if you call `EnqueueAsync()` many times or call `SaveChangesAsync()` several times inside one explicit transaction. Separate commits in the same scope each request readiness independently. In-memory enqueue callbacks finish before readiness is requested, so all committed messages are visible to the consumer. ## Failed saves and rolled back transactions Pending messages follow the same rules as your other pending changes on the context: - If `SaveChangesAsync()` fails, the messages remain pending together with the entity changes, and are saved by the next successful `SaveChangesAsync()`. - If `SaveChangesAsync()` succeeds inside a transaction that you later roll back, the messages are discarded just like the saved entity changes. They are not saved again by a later `SaveChangesAsync()`. ## Enqueueing outside an Entity Framework unit of work When an infrastructure entry point owns a standalone enqueue operation and must persist it before returning success, wrap it in an `ImmediateQueueOperationScope`: ```cs using (new ImmediateQueueOperationScope()) { await enqueuer.EnqueueAsync(message); } ``` Nexus uses this scope internally for API and MCP enqueue endpoints, incoming broker consumers, and hybrid-storage transitions. Application code should normally let enqueue operations join its scoped `DbContext` instead. ## Which queues join the unit of work Messages enqueued to database queues and in-memory queues join the Entity Framework unit of work: - Database queue messages are saved in the same transaction as your entity changes. - In-memory queue messages are added to the queue after the transaction has been committed. They are discarded together with the entity changes if the transaction is rolled back or the context is never saved. Messages that leave the application can't be part of a database transaction, so these are sent immediately and are not undone if your Entity Framework transaction is rolled back: - Messages enqueued with `IApiEnqueuer` to Nexus running in another application. - Messages enqueued to queues using [external pending storage](external-pending.md), such as Azure Service Bus, RabbitMQ, Kafka or Redis. This includes messages that are stored in the database until they're ready to be processed, so that the queue behaves the same no matter which status a message is enqueued with. If such a message must only be sent when your entity changes are saved, use an outbox: enqueue the message to a database queue in the same unit of work, and forward it from there with a [queue processor](processors.md) or a queue job. Give the message an id and make it idempotent so that a retried forward doesn't create a duplicate. ## Only works with EnqueueAsync() Note that this sync with `SaveChangesAsync()` only works with `IEnqueuer.EnqueueAsync()` and not other Nexus services like `IQueueItemUpdater`. If you need that you can always create an explicit transaction yourself and use `DbConnectionScope` to make sure that any Nexus operations that happen in that scope use your explicit transaction. --- # External pending queue storage The great thing about Nexus is the control and insight you get into queue messages from using an SQL based queue storage. But for some cases the message throughput is just too high for it to make sense to use SQL based queues for everything. This is where external pending queue storage in Nexus comes in, specifically the `IExternalPendingStorage` interface. It lets you use Nexus without forcing you to store pending messages in an SQL-backed queue. Pending messages stay in Redis, RabbitMQ, Kafka, or Azure Service Bus. A queue job or distributed queue processor leases them from the processing channel; failures and other non-pending states move into the SQL-backed Nexus queue for inspection and management. This is very useful for cases like event ingestion where you get tons of messages/events but don't really care about individual messages when they're in the happy path of Pending -> getting processed -> Processed/deleted. But you do care about failures and want the insights you get from using Nexus. ## Idempotency, message id, etc There are some Nexus features that you may not be able to use with external pending queue storage such as idempotency, queue message history, messages with id getting deduplicated etc. How well these features can be supported depends on the feature set of the external queue storage. The Redis package supports idempotency, message ids and messages scheduled for future processing, so the Redis package is the best option if you're just looking to maximize message throughput and don't have any requirements on which underlying queue storage to use. ## Supported packages At the time of writing the Nexus.Redis, Nexus.RabbitMQ, Nexus.Kafka and Nexus.Azure packages offer external pending storage out-of-the-box. Use the extension method from the provider package, for example: ```cs builder.Services .AddNexus() .AddRedisExternalPendingStorage(options => { options.ConnectionString = redisConnectionString; }); ``` And in your queue message set it as external like this: ```cs [Queue("export_order", StorageMode = QueueStorageMode.ExternalPending)] public class ExportOrderQueueMessage : IQueueMessageWithId { } ``` ## Readiness bridges and direct producers Every external provider registration also starts an `IExternalQueueReadinessBridge`. The bridge observes a durable notification path, requests activation, and acknowledges that notification only after activation is durable. It does not lease or settle the processing message. A burst of notifications is coalesced by the configured activation store. Each registered bridge also contributes a Nexus health check named `NexusExternalQueueReadiness_{ProviderId}`. The check is unhealthy while a consumer has not started or while a provider connection, notification, or acknowledgement operation is failing, and returns to healthy after that queue's consumer succeeds again. Providers with no matching external-pending queues report healthy without opening a consumer. Producers that call a Nexus `IEnqueuer` automatically use the correct topology. Direct producers must use the same v3 topology: - **Redis:** store the envelope in `nexus:queue:{queue-name}` (or the configured prefix) and append the queue name to `nexus:queue:{queue-name}:readiness` in the same Lua script. Delayed messages go to the `:delayed` sorted set; the bridge promotes due entries and appends the readiness record atomically. The braces are required so the keys share a Redis Cluster slot. - **RabbitMQ:** publish to the fan-out exchange `{ExchangeNamePrefix}{queue-name}`. Nexus binds both `{QueueNamePrefix}{queue-name}` and its configured readiness-suffix queue to that exchange. Do not publish directly to the processing queue. - **Kafka:** publish to `{TopicNamePrefix}{queue-name}`. The processing and readiness consumers use different consumer groups, so readiness commits never advance processing offsets. - **Azure Service Bus:** publish to the topic `{QueueNamePrefix}{queue-name}`. Nexus receives from the configured `processing` and `readiness` subscriptions independently. Do not publish to a legacy queue entity. The broker message body/member must use the Nexus external-pending envelope. `MessageJson` is the serialized queue-message JSON, not an escaped object supplied separately by the broker: ```json { "MessageJson": "{\"OrderId\":\"123\"}", "MessageId": "123", "Priority": 0 } ``` For Redis, an immediately ready direct enqueue must add that complete envelope to the ready sorted set and append one readiness record atomically. With the default prefix, all keys use the same Redis Cluster hash tag: ```lua -- KEYS[1] = nexus:queue:{queue-name} -- KEYS[2] = nexus:queue:{queue-name}:readiness -- ARGV[1] = priority, ARGV[2] = envelope JSON, ARGV[3] = logical queue name redis.call('ZADD', KEYS[1], ARGV[1], ARGV[2]) redis.call('XADD', KEYS[2], '*', 'queue', ARGV[3]) return 1 ``` For a batch, add every envelope and append only one readiness record after at least one ready message was added. Delayed envelopes belong in `{KeyPrefix}{queue-name}:delayed` with their Unix timestamp in seconds as the score; the readiness bridge atomically promotes due values and publishes the readiness record. Direct producers that need Nexus's Redis message-ID replacement or idempotency semantics should use `IEnqueuer` rather than the minimal script above. These are breaking topology changes for RabbitMQ and Azure Service Bus, and Redis now uses different physical keys with cluster hash tags. Drain or explicitly migrate old resources before cutover; Nexus database migrations do not move broker messages. Create the v3 topology and switch direct producers before enabling readiness-driven processing. Centralized reconciliation recovers missed readiness notifications, but it cannot make publications to an old physical queue observable by the bridge. ### Using multiple different external storages Note that you can use both eg Redis and RabbitMQ at the same time on different queues like this: ```cs builder.Services .AddNexus() .AddRedisExternalPendingStorage(options => options.ConnectionString = redisConnectionString) .AddRabbitMqExternalPendingStorage(options => options.ConnectionString = rabbitMqConnectionString); ``` ```cs [Queue("export_order", StorageMode = QueueStorageMode.ExternalPending, ExternalProvider = "redis")] public class ExportOrderQueueMessage : IQueueMessageWithId { } [Queue("import_order", StorageMode = QueueStorageMode.ExternalPending, ExternalProvider = "rabbitmq")] public class ImportOrderQueueMessage : IQueueMessageWithId { } ``` --- # Filterable message properties In some cases you might want to find messages where one or more property contains certain values. Like if you have a queue of inventory messages and you don't want to process it until the article data has come in. If your inventory message class looks like this: ```cs public class InventoryChangeQueueMessage : IQueueMessage { public required int ChangedQuantity { get; set; } public required string Sku { get; set; } } ``` When the message is processed we check if an article exists with that SKU and if it doesn't we want to skip processing it, but we don't want to delete it. We'll schedule it to be retried later, but we want to minimize the wait time after the article has been created and when we process its inventory messages. So when the article has been created we can update the status of messages by passing in a lambda like this: ```cs class ProductController(IQueueItemUpdater updater) { public async Task OnArticleCreatedAsync(string sku) { await updater.UpdateStatusWhereAsync(QueueItemStatus.Pending, m => m.Sku == sku); } } ``` The lambda is translated into SQL and by default it will use the JSON functionality in the database to filter by message property. If you have a lot of messages in the queue that can become a performance issue as it won't be able to use any indexes for the query. If you're concerned about performance you should use the `[Filterable]` attribute as described below. # Reading queue items by property If you have more advanced use cases than just updating status by message property you can use the `IQueueReader` to read queue items that matches a lambda filter. Eg: ```cs class ProductController(IQueueReader reader) { public async Task OnArticleCreatedAsync(string sku) { var items = await reader.ReadAsync(m => m.Sku == sku); foreach (var item in items) { // Do something interesting } } } ``` # [Filterable] attribute If you want to filter on message properties and you're concerned about performance you should add the `[Filterable]` attribute on your property like this: ```cs public class InventoryChangeQueueMessage : IQueueMessage { public required int ChangedQuantity { get; set; } [Filterable] public required string Sku { get; set; } } ``` This tells Nexus that you want enhanced filtering on the `Sku` property which means that we will extract the contents of that property into a separate database column that gets a database index to improve performance of filtering. The `[Filterable]` attribute is allowed on properties of primitive type (including `DateTime` and `Guid`) as well as lists of primitive types. Eg: ```cs public class InventoryChangeQueueMessage : IQueueMessage { public required int ChangedQuantity { get; set; } [Filterable] public required List Skus { get; set; } } ``` Nexus will create a separate table per list property in order to efficiently filter on values for it. Nexus will also automatically populate new filterable columns and tables from the messages in the queue when you've added `[Filterable]` attributes. This is done during initialization when the saved queue definition changes. Existing messages must remain readable by the new contract for backfills to succeed. Coordinate instances when changing/removing filterable properties; Nexus does not re-inspect or repair unchanged schemas after manual edits. See [database initialization](../general/database.md#database-initialization). Nexus automatically creates database indexes for these columns and tables, but won't combine multiple columns in these indexes. Which means that if you want to filter on multiple properties in a single query you might want to create a combined index on those columns. # Lambda limitations The lambda passed to `IQueueItemUpdater` and `IQueueReader` can combine and/or comparisons as well as grouping by parentheses such as `m => m.Sku == sku && (m.ChangedQuantity == 1 || m.ChangedQuantity == 2)`. The lambda can filter on deep properties such as `m => m.Some.Deep.Object == value` but note that the `[Filterable]` attribute only work on top-level properties on a message class. So deep properties like this will always use the less performant JSON operators in the database. The lambda can check if a list of primitive values contains a value or not, such as `m => m.ListOfSkus.Contains(sku)` or `m => !m.ListOfSkus.Contains(sku)` but no other comparisons on lists are allowed. Dictionaries are not supported for filtering. If you have a dictionary on your message and you can't convert that to a list you can create a separate filterable property that's derived from the dictionary instead: ```cs public class ArticleChangeQueueMessage : IQueueMessage { public required Dictionary Categories { get; set; } [Filterable] public List CategoryIds => Categories.Keys.ToList(); } ``` # Filtering in the Admin UI By default all properties with the `[Filterable]` attribute will be extra visible and filterable in the Admin UI. If you don't want a filterable property to show up in the Admin UI like that you can hide it by doing `[Filterable(VisibleInAdminUI = false)]`. # Filtering guarantees In order to guarantee consistency Nexus stores primitive filterable properties as columns in the queue table. Which means that you won't risk getting out-of-sync between the message itself and the filterable property column since it's committed to the database in the same query. In the case of list properties Nexus will populate the separate tables after the messages are inserted/updated in a queue table, but it's done inside a transaction to prevent inconsistency. If you're explicitly using an isolation level in the database that allows dirty reads (such as `read uncommitted` in SQL Server) you risk reading or updating the wrong messages. --- # Auto generated queues Generated definitions use the [named queue infrastructure](named.md). Messages remain instances of the original CLR type; Nexus no longer emits message or job subclasses. Configure `RoutingQueueName` to create an explicit virtual entry point for `ShouldEnqueue` routing, or enqueue directly into a concrete generated queue by name. Several libraries can contribute generated queues and jobs through separate `AddQueues()` and `AddJobs()` calls. Append to the shared options collections and call `Build()` once after all contributions. Each generated job's concrete queue must be declared before its `AddJobs()` call. Definitions and shared routing entry points are registered once at build, with duplicate-name and processing-conflict validation. See [library contributions](named.md#contributions-from-libraries). Sometimes you have messages with the same structure but one or more discriminator properties makes them significantly different than others. Different enough that you want such message split up into multiple different queues. An example might be a message that has a `Language` property and you want to have different queues per language so each language can be processed individually. Or if you want to enable `PauseOnError` so that any error for that language pauses processing for any other message in that language. It's possible to manually define a bunch of queues and jobs using inheritance to handle this, but it comes quite inelegant quickly because of the boilerplate needed. Generated definitions create concrete queue registrations and job bindings from configuration, using an ordinary message contract and job implementation. For example: ```cs // Program.cs [Queue("language")] public class LanguageQueueMessage : IQueueMessage { public string Language { get; set; } = ""; } public class LanguageQueueMessageJob : IQueueJob where TMessage : LanguageQueueMessage { public Task ProcessMessageAsync(TMessage message, QueueProcessingContext context, CancellationToken cancellationToken) { return Task.FromResult(ProcessResults.Processed()); } } var generatedQueues = new List(); var generatedJobs = new List(); foreach (var language in GetLanguages()) { var queueName = "language_" + language; generatedQueues.Add(new GeneratedQueue { RoutingQueueName = "languages", Name = queueName, DisplayName = "Language queue " + language, Category = "Language", ShouldEnqueue = message => message.Language == language, }); generatedJobs.Add(new GeneratedQueueJob { JobBaseType = typeof(LanguageQueueMessageJob<>), Name = "language_job_" + language, DisplayName = "Language job " + language, Category = "Language", QueueName = queueName, }); } builder.Services .AddNexus() .AddQueues(options => { options.GeneratedQueues = generatedQueues; }) .AddJobs(options => { options.GeneratedJobs = generatedJobs; }); ``` With this code you you'll get one concrete queue and job per language returned from the fictional `GetLanguages()` method. `ShouldEnqueue` is evaluated when enqueueing to the configured routing entry point, such as `enqueuer.EnqueueAsync("languages", message)`. Every matching destination receives the original message payload through the ordinary virtual queue pipeline. Direct enqueueing to `language_sv` bypasses group routing. Use an ordinary concrete job class or a concrete open generic implementation. Nexus closes generic implementations over the original message type. Abstract generated job implementations must be made concrete; no runtime derived job class is emitted. Explicit generated job names, schedules, and message filters remain part of the job binding metadata. Filters are configured expressions; parameters are not used to derive them. ## Enqueueing messages Generated destinations require explicit queue names, even when only one exists. For group routing, supply the configured virtual entry name. To obtain a queue-bound typed accessor, use `IQueueResolver.GetEnqueuer("language_sv")`. Type-only enqueueing does not broadcast or evaluate `ShouldEnqueue`. ## Other queue operations If you need any of the other services for a generated queue you'll need to request a list of the non-generic version and filter out based on queue name. So let's say that we want the Swedish queue reader from the language example at the top on this page. We'd do something like this: ```cs var queueReaders = serviceProvider.GetServices(); // Or IEnumerable in your constructor var language = "sv"; var swedishQueueReader = queueReaders.Single(q => q.QueueName == "language_" + language); ``` --- # Queue message history Nexus can keep track of the history of messages with [identity](./identity.md). This means that you get a full audit trail of the content of the messages, when they change and what changed on them. Message history is disabled by default in Nexus since the history can grow a lot which can become costly. You can enable history on all queues where the queue message implements `IQueueMessageWithId` by configuring it when initializing the queue system: ```cs builder.Services.AddNexus().AddQueues(options => { // History will be stored forever on all queues with ids options.DefaultHistoryRetention = TimeSpan.MaxValue; }); ``` This will set the default retention which can then be overriden per queue using the `[Queue]` attribute: ```cs [Queue("myqueue", HistoryRetention = "30.00:00:00")] ``` The above will store the history for that queue for 30 days. The string is a `TimeSpan` in string format but also supports the special `forever` value to represent `TimeSpan.MaxValue`. ## Disabling history globally `QueueSystemOptions.DisableHistory = true` overrides both code defaults and history retention saved through the Admin UI/API. It prevents history recording and creation of history tables, and removes existing history tables during initialization. Leave it false if you want to enable history on individual queues. Retention cleanup runs in bounded batches; see [queue maintenance](maintenance.md). ## Accessing the history You can access the history in the [admin UI](../admin-ui/index.md) when going into a queue and selecting a queue message. The button to show message history will only be visible for queues with identity and where the history retention is not set to `TimeSpan.MinValue`. You can also access the message history using the API endpoint `GET /api/queues/:queueName/:messageId/history`. --- # Message identity and IQueueMessageWithId The `IQueueMessage` interface doesn't contain any properties so you can define your message in any way you want. But using `IQueueMessage` means that the system won't have any knowledge of message identity. The system won't compare two messages to see if they are identical to prevent duplicates in a queue. If you need to ensure that a queue never contains duplicates of the same message you should instead use the `IQueueMessageWithId` interface. The only property that you're forced to have on your message class is the `Id` property. What the id should be depends on your queue. If you want you can use a combination of values in the implementation of the property. The property is allowed to return `null` so you can have messages in the same queue where some have identity and others don't. If we take the `SendOrderConfirmationQueueMessage` from the example in the [Overview](./index.md) we can define it like this instead: ```cs [Queue("send_order_confirmation")] public class SendOrderConfirmationQueueMessage : IQueueMessageWithId { public string? Id => OrderId; public string OrderId { get; set; } public string Email { get; set; } } ``` By doing this the system will ensure that if we already have a `SendOrderConfirmationQueueMessage` message for order with id `123` we will never allow duplicates to be enqueued. If someone enqueues it again we'll instead update the existing message rather than adding another one. This is true regardless of which status the message has. Other queue systems has a separate deadletter queue where failed messages are placed which can allow duplicates to still happen. But if we have a message for order `123` in the `Error` status and someone enqueues a message with that id the existing message gets updated and the status is set to `Pending` again. Read more about message statuses [here](./index.md#message-status). ## Enqueueing while a message is leased For database and in-memory queues, enqueueing a new version of a leased message sets its status to `AwaitingRelease`. The current processor keeps its original lease, and no other processor can lease the newer version yet. The latest enqueue supplies the payload, priority, processing date, and `StatusAfterRelease`. Further enqueues replace those deferred values. When the current processor releases or finishes the message, Nexus applies that destination. Immediate messages become `Pending`, future messages remain `Sleeping`, and custom destinations retain their custom status. If a sleeping message's processing date has already passed, normal scheduled-message readiness rules apply immediately. The older processor's payload updates, priority, rescheduling, or failure status do not overwrite the newer enqueue. Successful completion still records the processed version as the previous message, even when processed-message retention is zero. If the lease expires instead, normal abandonment and `RetryAbandonedMessages` rules apply. This behavior does not apply to external pending storage, which retains its broker-specific delivery behavior. ## Idempotent messages If you use message identity you might also want to use [message idempotency](./index.md#idempotent-messages). It tells Nexus to discard incoming messages where the message property valies are exactly the same as a message with the same id in the queue. Which is handy if you only want to process changes but the queue is populated with data from a system that only allows full exports and can't tell you which messages/entities have changed since the last sync. --- # In-memory queues Queues in Nexus are backed by a database but in some cases you want to squeeze the last bit of performance out of your queues and then the overhead of a database can become too much for certain use cases, or if you need to relieve your database of the work needed to handle some very busy queues. For those scenarios Nexus lets you have in-memory queues without the overhead of reading and writing to a database first. Note however that this _only_ works if you only deploy Nexus to a single server. See the section below about single-server use. You can however allow other servers to enqueue but only if you use `IApiEnqueuer` to enqueue through the API to the single server hosting the API and the memory queue(s). ## Creating a memory queue You create a memory queue just like any other queue by creating a class that implements `IQueueMessage` or `IQueueMessageWithId`: ```cs [Queue("example_memory", StorageMode = QueueStorageMode.InMemory)] public class ExampleMemoryQueueMessage : IQueueMessage { // Properties here } ``` When you set `StorageMode = QueueStorageMode.InMemory` in the attribute it gets registered as a memory queue. Note that you can switch back and forth between having a queue as an in-memory queue or not because an in-memory queue will use its database queue as fallback storage to ensure that messages aren't lost because of deploy or application restarts. Messages and message changes are scheduled to be persisted to the database as soon as possible, but it's done on a background thread. The in-memory queue registers a listener for application shutdown to prevent the application from shutting down before all changes are persisted. ## Durability Nexus persists in-memory changes to fallback storage in the background so persisted changes survive restarts. An abrupt process failure can lose changes that have not been flushed. Fallback persistence is asynchronous and is not a durable commit before `EnqueueAsync()` returns. Nexus flushes pending changes during graceful shutdown, but abrupt termination, persistence failures, or an insufficient shutdown timeout can lose unflushed data. Use database queues when the producer needs a durable database write before returning. ## Fallback storage The in-memory queue will fallback to it's own regular database table by default which makes the transition from a non-memory queue to an in-memory queue smooth. But if you want to use an in-memory queue to lessen the load on your database it doesn't have the intended effect. In that case you can explicitly use the Sqlite driver and use Sqlite tables as fallback storage by installing the `CommerceMind.Nexus.Sqlite` NuGet package and adding this in your `Program.cs`: ```cs builder.Services .AddNexus() .AddSqliteMemoryQueueFallbackStorage(); ``` If using disk as fallback storage isn't an option for you it's possible to create your own implementation of the interface `IMemoryQueueFallbackStorage` and register it in the service collection. ## Single server use _only_ In-memory queues can generally only be used if you only have a single server/instance running Nexus. Even if changes to an in-memory queue is persisted to storage Nexus won't read from the storage except during initialization. You can use in-memory queues in a multi-instance application if the queues are local to that instance and messages in the same queue have no effect on messages in other instances in terms of sort order or processing side effects. In this case however you must implement your own `IMemoryQueueFallbackStorage` as the built in implementation uses a memory counter to increment assign queue item ids. If you're using a hosting service such as Azure App Service you probably can't use in-memory queues unless you make sure to stop the application before starting a deploy as the default in Azure App Service is to bring up a parallel deployment slot and swap with the live slot. Which means that during deploy you will have multiple instances running. If this is an issue for your application will depend on how you use the queues. ## Application shutdown In-memory queues registers a callback on the applications stopping token (`CancellationToken`) and performs a synchronous flush of all pending changes to the fallback storage. Nexus will also attempt to stop all jobs when shutdown starts and won't start any jobs after that. But it's up to you not to enqueue to memory queues after shutdown has started. If you do you will risk enqueueing a message that doesn't have time to be persisted before the process exits. ## In-memory queues and transactions In-memory queues does not offer a way to integrate with the transaction support in `DbConnectionScope`. Outside the Entity Framework integration, in-memory changes are visible immediately to the whole application, and changes are sent to the fallback storage explicitly without transaction support to not accidentally enlist those queries in unrelated transactions. With the [Entity Framework integration](entity-framework.md), enqueue operations wait for the scoped context's asynchronous commit. Committed messages are added to memory before readiness is requested; rolled-back or unsaved operations are discarded. This does not make fallback persistence synchronous. Other in-memory updates still do not roll back with a database transaction. For explicit transactions outside the EF integration, publish to memory only after the transaction commits, or use a database outbox queue if the handoff must be durable. ## Processing and activation In-memory queues support readiness-driven or scheduled [queue jobs](jobs.md), including the same [leased-message replacement](identity.md#enqueueing-while-a-message-is-leased) rules as database queues. They do not support distributed queue processors. In-memory queue storage and [in-memory activation storage](activation.md#activation-storage) are separate choices; selecting one does not select the other. --- # Queues Unlike popular message buses and event queues this system uses an SQL database to store messages and also offers [in-memory storage](./in-memory.md). Message bus systems are typically built to be able to handle tons of messages and have the assumption that you're not really interested in looking at a single message and doesn't offer great support for finding and manipulating messages. Nexus on the other hand assumes that you won't have millions of messages per second and that you are interested in looking at individual messages. And for that an SQL database is a perfect fit. At the time of writing we support Postgres, SQL Server and SQLite. Feel free to reach out to us if you need support for another database engine like MySQL or MariaDB. Each registered queue is its own database table which is automatically created by the system when you've registered your queue. And a message is a row in that database table which contains the message in JSON format together with meta data such as the status and when it was created. ## When SQL based queues aren't enough If you get more incoming messages than what a SQL based queue can handle in a performant way you should look into [external pending storage](./external-pending.md) for Nexus queues. It gives you the best of both worlds - the insights and troubleshooting capabilities from Nexus with the raw throughput of queue systems like RabbitMQ, Redis, etc. ## Defining and registering a queue The first thing to do is to create a C# class representing the message type of the queue. It's just a .NET class implementing the interface [`IQueueMessage`](https://github.com/Commerce-Mind-Nexus/Nexus/blob/main/src/job-engine/Nexus.Abstractions/Queueing/IQueueMessage.cs). Here's an example: ```cs [Queue("send_order_confirmation")] public class SendOrderConfirmationQueueMessage : IQueueMessage { public string OrderId { get; set; } public string Email { get; set; } } ``` You can add any properties you want on the class, the only important thing to remember is that the message will get serialized to and from JSON. There's also the `IQueueMessageWithId` interface which you can read about in the section about [message identity](./identity.md). ### [Queue] attribute The queue attribute isn't strictly required but it's recommended as it sets meta data for the queue such as the queue name. That name is used for naming the database table and to access the queue in the API and admin UI. If the queue attribute isn't set the name is taken from the class name. If you don't use the `[Queue]` attribute it's easy to accidentally rename the class which will create a completely new and empty queue. The `[Queue]` attribute also lets you set other types of meta data such as storage mode, activation/processing delays, and processed-message retention. ### Registering the queue Just like scheduled jobs the queues are automatically registered by finding all types that implements [`IQueueMessage`](https://github.com/Commerce-Mind-Nexus/Nexus/blob/main/src/job-engine/Nexus.Abstractions/Queueing/IQueueMessage.cs) (`IQueueMessageWithId` inherits from `IQueueMessage`). And just like for jobs this is only done automatically for messages defined in the entry assembly. If you have multiple .NET projects that contains queue message classes you need to register the assemblies with the service. There's a property on the options object passed into `AddNexus().AddQueues()` called `AssembliesToScanForQueueMessages` where you can add additional assemblies to scan for queue messages like this: ```cs builder.Services.AddNexus().AddQueues(options => { options.AssembliesToScanForQueueMessages.Add(typeof(MyQueueMessageInAnotherProject).Assembly); }); ``` ## Choosing a processing model Register a [queue job](jobs.md) with `AddJobs()` when you want one active run across instances, job history/parameters, mutexes, or a cron schedule. Register a [distributed queue processor](processors.md) with `AddQueueProcessors()` when independent messages should be handled by worker pools on every instance. A concrete queue can have one processing job or one distributed processor. Registering queues alone does not enable either. [Named queues](named.md) let several independent queues share one message type. [Generated queues](generated.md) use that same infrastructure with optional explicit routing. Named/generated destinations require a queue name; type-only enqueueing does not choose or broadcast to them. ## Adding a message to a queue There's two ways to add queue messages. The first is through the C# SDK using the [`IEnqueuer`](https://github.com/Commerce-Mind-Nexus/Nexus/blob/main/src/job-engine/Nexus.Abstractions/Queueing/IEnqueuer.cs) service. That service is injectable through the `IServiceProvider` so you can take it as a constructor dependency. The generic argument `TMessage` is the message type that you want to add a message for. An example: ```cs public class OrderController { private readonly IEnqueuer _enqueuer; public OrderController(IEnqueuer enqueuer) { _enqueuer = enqueuer; } [HttpPost] public async Task OrderPlacedAsync(string orderId, string email) { await _enqueuer.EnqueueAsync(new SendOrderConfirmationQueueMessage { OrderId = orderId, Email = email, }); return NoContent(); } } ``` With the default enqueue status and no processing delay, the message becomes `Pending`. Queue jobs without a saved schedule are readiness-driven, so `EnqueueAsync()` requests [queue activation](activation.md) and processing can start as soon as the operation commits. If the job is disabled, the activation remains pending and processing resumes after it is enabled. Give the queue job a cron schedule when deliberate batching, rate limiting, or maintenance windows require scheduled-only processing. There's also the non-generic version of `IEnqueuer` which lets you enqueue any object of type `IQueueMessage`. The non-generic `IEnqueuer` will find the correct enqueuer for the messages you pass in and send it to the correct queue. ### Adding a message through the API The other way of adding a message to a queue is by using the API. The API is just an way to access the same [`IEnqueuer`](https://github.com/Commerce-Mind-Nexus/Nexus/blob/main/src/job-engine/Nexus.Abstractions/Queueing/IEnqueuer.cs) over HTTP at the endpoint `POST /api/queues/{queueName}` relative to the API base path. ### Adding multiple messages at once Both the C# SDK and the API allows you to enqueue multiple messages at the same time and will use batch operations (except SQLite which doesn't have batch imports) to make enqueuing a lot messages as fast as possible. ## Message status Re-enqueueing a leased message with identity sets it to `AwaitingRelease` until its current processor finishes. The latest enqueue's destination is stored separately in `StatusAfterRelease`. See [enqueueing while leased](identity.md#enqueueing-while-a-message-is-leased). A message has a status and the default status is `Pending`. It means that the message is up-for-grabs for a queue processing job. When a queue processing job starts it will see if there's any `Pending` messages and then try to lease them. This is done by using the [`IQueueReader.LeaseAsync()`](https://github.com/Commerce-Mind-Nexus/Nexus/blob/main/src/job-engine/Nexus.Abstractions/Queueing/IQueueReader.cs) method which will update the status of any `Pending` messages to `Leased` together with setting the date for how long they've been leased for. Leasing a message prevents other processors from selecting that message until the lease expires or is released. A processor still running after lease expiry can overlap with a retry, so choose lease times for the full expected processing duration. If the lease expires - that is, if the processing job hasn't finished in the time it set for the lease - the message status till be set to `Abandoned`. When this happens it most likely means that the job that leased the message crashed and won't recover. When a message is set as `Abandoned` you need to manually look into what happened and decide if you want to retry the message by changing its status back to `Pending` or if you want to delete the message and handle the work in some other way. In the Admin UI it's possible to configure a queue (under the More button on the queue details page) to retry messages when the lease expires instead of setting them to `Abandoned`. This should only be enabled for messages where it's safe to process the same message twice. The message might have been processed once, but the server crashed before Nexus was able to update the status to `Processed`. When a queue processing job is done with the message the status will either become `Error` or `Processed`. And if the queue is not configured to keep processed messages the message will be deleted rather than set to `Processed`. The system also allows you to set custom statuses on messages through the UI and the API which allows you to categorize messages for troubleshooting as well as prevent them from being processed by the system. Messages with custom statuses are invisible to the processing system and are only visible in the UI and API. ## Updating status or reading by property in message Sometimes you might want to update the status of messages where a certain property inside messages contains a specific value. An example would be that you have a queue of inventory messages and you don't want to process it until the product data has come in. So your queue processing job for the inventory messages keeps rescheduling such messages. When the product data then comes in you can update the inventory message(s) like this: ```cs public async Task OnProductCreated(string sku) { await _inventoryQueueItemUpdater.UpdateStatusWhereAsync(QueueItemStatus.Pending, m => m.Product.Sku == sku); } ``` See more [in the docs about filterable properties](./filterable.md). ## Default enqueue status As mentioned above the default enqueue status is `Pending` which makes the message immediately available for processing jobs. In some cases you might want to change this to a different status so that new messages that comes in are kept in a different status and becomes invisible to processing jobs. You can do this in the Admin UI for the queue or through the API and you can use any text value you want such as `Paused` or `OnHold`. This can be useful if a queue processing job is experiencing problems and you only want to run it on selected messages. Then you can change the default enqueue status and manually change only the messages you want to process to `Pending`. After the issues have been resolved you can filter out all the messages with status `Paused` or `OnHold` and update to `Pending` again. ## Processing delay Sometimes you want to wait a bit before processing a message. Maybe you need to make sure that you only process the last message of it's kind and for that you need to wait a while to see if a new one comes in before you process it. Or maybe you need a delay to give some other system time to prepare first before you send them data. You can control a delay on message processing either on the `[Queue]` attribute using eg `[Queue("myqueue", ProcessingDelay = "30m")]` or you can set it on-the-fly in the Admin UI. ## Previous message version For messages with [identity](identity.md), Nexus can retain the previously successfully processed version alongside a new version. This is useful for doing change based processing since it allows you to compare the current message with the previous message and add additional processing logic when certain properties of the message changes. In order to access the previous message from inside a [queue processing job](./jobs.md) you use the `IPreviousMessageProvider` service. It is available during built-in queue-job and distributed-processor handling, and returns `null` outside an active processing context or when no previous version exists. If you need access to previous messages outside of a job you can read `QueueItem`s using `IQueueReader`. `QueueItem` contains everything about the queue item including the current and previous message. :::note[Previous version] The previous version is only updated when calling `IQueueItemUpdater.ProcessedAsync()` which is done for you by built-in queue jobs and distributed processors. It's then updated to be the message passed to that method. This ensures that the previous version of a message will only ever be a version that was successfully processed. If the message processing fails the previous version stays untouched to let you retry the message. ::: ## Idempotent messages If messages are fully self-contained, meaning that a processing job will only use data on the message itself to perform it's processing and not look at any external data your messages are idempotent. Processing an idempotent message will always result in the same effect. For the queues you have which has this characteristic you can tell the `[Queue]` attribute that it's idempotent like this: ```cs [Queue("product_updates", IdempotentMessages = true)] public class ProductUpdatedQueueMessage : IQueueMessageWithId { ... } ``` When `IdempotentMessages` is set to `true` it means that if a message is enqueued Nexus will check if we already have that message with the same message id and JSON and then discard the message. Nexus will compare the full message JSON by default, but if your message contains timestamps or other things that doesn't matter from an idempotency perspective you can have a specific property be the idempotency key. If you set the `[IdempotencyKey]` attribute on the property then Nexus will use that for deduplication instead of the whole message. This is useful for big data syncs such as a product catalog. You can have a job that reads the entire product catalog from another system and enqueues a message per product. If nothing has changed on the product since the last time it was enqueued the message won't get processed again. The comparison is done at a JSON string level so a message will only be discarded if the JSON is exactly the same. Which means that the order of properties in the serialized message is important. This is only something you need to consider if your message contains dynamic collections such as dictionaries where the order can change. :::note[Idempotency and processed messages] When you set `IdempotentMessages` to `true` the `ProcessedMessagesRetention` property is automatically set to never delete processed messages unless you explicitly specify a timespan to keep processed messages for in the queue. ::: ## Partial messages Sometimes you might have multiple data sources that are each responsible for their own section of a message. Such as in an e-commerce context where the ERP might be responsible for the price and the PIM is responsible for the product data, but you want to process the data from these systems in a single message. In such a case you can use partial messages in Nexus, which is message classes where the top level properties are set by different enqueuers. Take this message for example: ```cs public class ProductUpdatedMessage : IQueueMessageWithId { public string? Id { get; set; } public ProductInformation? ProductInformation { get; set; } public ProductPrice? Price { get; set; } } ``` When listening to events from the PIM you do this: ```cs public class ProductInformationController(IEnqueuer enqueuer) : Controller { [HttpPost("{productNumber}")] public async Task ProductUpdatedAsync(string productNumber, [FromBody] ProductInformation productInformation) { await enqueuer.EnqueueAsync(new ProductUpdatedMessage { Id = productNumber, ProductInformation = productInformation }, new EnqueueContext { PartialMessage = true }); return Ok(); } } ``` And when listening for ERP price updates you do: ```cs public class PriceController(IEnqueuer enqueuer) : Controller { [HttpPost("{productNumber}/price")] public async Task PriceUpdatedAsync(string productNumber, [FromBody] ProductPrice price) { await enqueuer.EnqueueAsync(new ProductUpdatedMessage { Id = productNumber, Price = price }, new EnqueueContext { PartialMessage = true }); return Ok(); } } ``` What will happen in the background here is that Nexus will merge the incoming message with an existing message with the same id. Meaning that when the PIM data comes in it won't remove the price from the message and vice versa. If there's no existing message with the same id a new message with be enqueued with only that data. If the processing job needs both in order to correctly process the product it can just return a custom status or just return that it processed the message. The message will get the `Pending` status again as soon as any of the systems involved have sent their data. A word of caution about the values `false`, `0`, `null` together with partial messages is that Nexus will ignore any properties that have default values. This because Nexus can't know if you explicitly set a property to a default value or if the property hasn't gotten a value and should be ignored. So you can't use partial messages to update eg a boolean property to `false` because Nexus won't know if you explicitly set the value to false or if that was just the default value. The recommendation is to use nullable types or enums instead. :::note[Only top level properties are merged] When using partial messages Nexus won't do a deep, recursive merge. Which means that if you have a message that looks like this: `{"Key1": {"DeepKey1": "deepvalue1"}, "Key2": "value2"}` and enqueue a partial message that looks like this: `{"Key1": {"DeepKey2": "deepvalue2"}, "Key3": "value3"}` the end result will be: `{"Key1": {"DeepKey2": "deepvalue2"}, "Key2": "value2", "Key3": "value3"}`. That is, `Key1` is replaced and `Key1.DeepKey1` was removed. Note however that top level dictionaries are supported, so if `Key1` on your message is a `Dictionary` it would instead be merged into: `{"Key1": {"DeepKey1": "deepvalue1", "DeepKey2": "deepvalue2"}, "Key2": "value2", "Key3": "value3"}`. Top level dictionaries can have any type and not just strings as in this example. ::: ## Dynamic message content Sometimes you get messages from external sources that have varying structure depending on what type of message it is but you don't want to create one Nexus queue per message type. And you might not want to create a ton of different properties on your message class where only some of them are relevant together. In such cases you can let your message implement `IDynamicQueueMessage`. This adds a property called `MessageData` to your message which represents the JSON that will be serialized. Which means that the `MessageData` property holds the raw JSON data and you can have other properties read from that. Something like this: ```cs class MyDynamicQueueMessage : IDynamicQueueMessage { public JsonElement MessageData { get; set; } public string? Type => MessageData.GetProperty("type").GetString(); public IMessageContent Content => Type == "FirstType" ? MessageData.Deserialize() : MessageData.Deserialize(); } public interface IMessageContent { ... } public class FirstTypeContent : IMessageContent { ... } public class SecondTypeContent : IMessageContent { ... } ``` This lets you use the `MyDynamicQueueMessage` class in a more convenient way as you don't have to include all the properties from all the different types. This also allows you to store data in the messages that aren't defined in the message class which can come in handy if an external system defines the message structure and you want to make sure that you always store exactly what they send you. When using `IDynamicQueueMessage` Nexus will only serialize and deserialize the `MessageData` property, all other properties on your message is ignored. If you want to use [messages with identity](./identity.md) with dynamic messages you simply let your message class implement both `IDynamicQueueMessage` and `IQueueMessageWithId`. Note that if you're using [Json.NET](https://www.newtonsoft.com/json) it still works, but Json.NET won't be serializing or deserializing a `IDynamicQueueMessage` even when you're using the `CommerceMind.Nexus.NewtonsoftJson` NuGet package. If you need Json.NET specific converters you'll need to first convert to/from JSON with Json.NET before passing it to `IDynamicQueueMessage`. ## Scheduling messages for future processing There's an additional status called `Sleeping` that messages that should be processed in the future will have. When you enqueue a message you can specify a `DateTime` or a `TimeSpan` (relative to right now) for when a message should be processed. This is useful for scenarios where you know that something will happen at a certain time in the future and you want to run some processing at that time. A typical e-commerce scenario is that a product can have prices with a start and stop date. You want to export your prices to another system but that system might not support date based validity for prices so you need to export the current price at any given time. To deal with this you can enqueue messages that should get processed at the time that one price stops being active and another starts. And your queue processing job can send the new price to the external system exactly when it starts being valid. ### Processing the same message again in the future Sometimes when you're using [idempotent messages](./index.md#idempotent-messages) to update external systems you might have dates in your message that affects the outcome of the processing. Such as a product message which contains a list of prices that have a valid from and to date. At the time of processing you pick the price that is valid right now, but you want to run the processing again when the next price becomes valid. In your queue processing job you can return an optional `DateTime` for when to process the message again. See more details in the section about [queue jobs](jobs.md#scheduling-a-processed-message-to-be-processed-again). ## Example queue messages The admin UI needs to create an example message to prefill the text box where you can manually enqueue a new message. By default Nexus will try to create an instance of your queue message class with default values in properties. If you want to customize that message, you can add a static method on your message class called `GetExampleMessage` like this: ```cs private class MessageWithExampleMethod : IQueueMessage { public string? SomeProperty { get; set; } public static IQueueMessage? GetExampleMessage() { return new MessageWithExampleMethod { SomeProperty = "From example" }; } } ``` If you want even more control over this you can register your own implementation of the interface `IExampleQueueMessageFactory` which is the service that will be called to create these example messages. ## Processing settings The new Admin UI can override threads/workers, lease time, lease size, and batch size per concrete queue where supported. Saved overrides take precedence over code defaults. See [queue processing settings](processing-settings.md) for update and reset behavior. ## Leasing messages Built-in processing leases messages before handling them so other workers cannot select them during the active lease. And Nexus ensures that if two processes try to lease the same message only one of them will win. The default lease time is 30 minutes. If you have a batch job with a very large batch size and the job takes a long time to complete you might want to increase the lease time for that job. You do that by implementing the `MessageLeaseTime` property from `IQueueJobBase`, like this: ```cs public class ExampleQueueJob : IQueueJob { public TimeSpan MessageLeaseTime => TimeSpan.FromMinutes(60); } ``` ### Abandoned messages If a server crashes in the middle of processing messages the lease will eventually expire. Note that this only happens in extraordinary cases. Exceptions that are thrown by the processing job are caught by Nexus and correctly transitions the message status from `Leased` to `Error`. But there can be a power outage or the database server might become unavailable which are things that Nexus can't automatically handle. In such cases the lease will eventually expire and at that point Nexus will transition the message status from `Leased` to `Abandoned`. If you see that any of your queues contains abandoned messages you need to investigate what happened first before resetting the status to run them again. It could be that the messages were fully processed but the server or database crashed right before Nexus was able to update the status to `Processed`. The best way to start investigating is by looking at the correlation id set in the messages. That is the correlation ID of the job run or distributed processing chunk that leased it which means that you can use the Admin UI or your log UI of choice to try to find out how far the job with that correlation id got. If the correlation id is "123" your logs related to that job run would have a log property called `Nexus.CorrelationId` with the value `123`. ## Health status All queues automatically get a health check and the default behavior is to mark the queue as unhealthy when there's a message with the `Error` status in it. However that's not always what you want because you can have qeueues for systems that you know have intermittent failures and you don't want the health check monitoring to keep bugging you about it. In that case you can either disable the health check for the queue or you can change which health status the queue has when it has messages of `Error` status in it, like this: ```cs [Queue("myqueue", HealthStatusWhenHasErrors = HealthStatus.Healthy)] private class MyQueueMessage : IQueueMessage { ... } ``` ## Organizing the Admin UI If you have a lot of queues you can group queues in the Admin UI by a category, just like you can with jobs. Set a category in the `Queue` attribute like this: ```cs [Queue("myqueue", Category = "My category")] private class MyQueueMessage : IQueueMessage { ... } ``` --- # Queue processing jobs There's two interfaces for queue processing jobs included in the SDK. One implementation that process messages one by one and one that process messages in batch. ## Accessing the previous message Queue jobs and distributed processors can both read the current and previously successfully processed version of a message. Such as a message for product information updates where you want to compare the previous and the current message to know what changed. You access it using the `IPreviousMessageProvider` service from inside your job. Read more [here](./index.md#previous-message-version). Queue jobs run once at a time across the cluster. For worker pools on every instance, see [distributed processors](processors.md). [Queue activation](activation.md) explains readiness-driven versus scheduled-only starts. ## Processing messages one by one This is the easiest and safest way to process a message since processing one by one never risks missing a message, it will either complete or fail. You implement such a job by creating a class that implements [`IQueueJob`](https://github.com/Commerce-Mind-Nexus/Nexus/blob/main/src/job-engine/Nexus.Abstractions/Jobs/Queueing/IQueueJob.cs). The job executor takes care of leasing messages and all you have to do is implement `ProcessMessageAsync()`: ```cs public class ExampleQueueJob : IQueueJob { public async Task ProcessMessageAsync(ExampleQueueMessage message, CancellationToken cancellationToken) { await DoSomethingInteresting(message); return ProcessResults.Completed(); } } ``` The job will run until there are no more `Pending` messages in the queue and will then return a short summary of how many messages that got processed and how many of them that failed as the job results. Note that even though the `ProcessResults` class has a `Failed()` method you don't have to catch exceptions yourself. The executor will catch any exceptions that happen in `ProcessMessageAsync()` and mark that message with the `Error` status together with the exception details in its processing log. ## Updating message during processing Sometimes you want to store meta data about the processing of a message and use that meta data the next time you process it. An example being syncing entities to another system and then storing the id the entity got in the other system so the next time we process the message we can use that id when communicating with the external system. Or you want to store error details that the next run can use when retrying a message. This can be achieved by returning an updated message as part of the returned `ProcessResult`. Eg: ```cs public class ExampleQueueJob(ILogger logger) : IQueueJob { public async Task ProcessMessageAsync(ExampleQueueMessage message, CancellationToken cancellationToken) { var id = await SendToExternalSystemAsync(message); message.IdInExternalSystem = id; return ProcessResults.Processed(updatedMessage: message); } } ``` Note that if a new version of the message is enqueued the default behavior is to replace the message, which means that your updated message with the id is cleared. You can work around that by using [partial messages](./index.md#partial-messages) since that won't replace the message, only replace the properties that the new message contains. ## Pausing on error If you have a queue and a job where the processing of a message is dependent on that previous messages have been successfully processed you can tell Nexus to pause processing when an error occurs. Nexus will by default process messages in the order they are created, but will continue to process the next message if a previous message fails. To pause processing when an error happens, set `PauseOnError` to `true` in the `[Job]` attribute on your job. This means that as soon as any message has either `Error` or `AwaitingRetry` status in the queue the processing will be paused and won't continue until those messages gets another status. Use this with caution as you risk pausing processing indefinitely unless you monitor the queue for errors. ```cs [Job(PauseOnError = true)] public class ExampleQueueJob(ILogger logger) : IQueueJob { public async Task ProcessMessageAsync(ExampleQueueMessage message, CancellationToken cancellationToken) { // Important to wrap the entire job in a try-catch with a retry. Otherwise you risk exceptions // stopping the queue processing until someone notices it. try { await DoSomethingInteresting(message); return ProcessResults.Processed(); } catch (Exception e) { logger.LogError(e, "An error occured"); return ProcessResults.RetryLater(TimeSpan.FromMinutes(10)); } } } ``` Note that this can be overriden in the Admin UI/API under the More button in the job details page. ## Processing messages in batches The other interface is [`IBatchQueueJob`](https://github.com/Commerce-Mind-Nexus/Nexus/blob/main/src/job-engine/Nexus.Abstractions/Jobs/Queueing/IBatchQueueJob.cs), where you're passed a batch of messages instead of getting them one by one. When you're creating a queue processing job that will read data from an external system it's often more efficient to batch load such data rather than requesting it one by one. Such as having a queue for when product data is updated and your job needs to read the full product object. Processing messages in a batch is a bit harder to do correctly than processing one by one, but the executor will take care of most of the complexity for you. Here's an example implementation: ```cs public class ExampleQueueJob : IBatchQueueJob { public async Task ProcessMessagesAsync(QueueMessageBatch batch, CancellationToken cancellationToken) { foreach (var message in batch.Messages) { cancellationToken.ThrowIfCancellationRequested(); try { await DoSomethingInteresting(message); batch.Processed(message); } catch (Exception e) { batch.Failed(message, e); } } } } ``` You are responsible for letting the system know which messages you've completed and which failed. If you didn't have the `try-catch` and there was an exception when processing the fifth message in a batch of ten the system will mark all messages as failed that hasn't explicitly been passed to `batch.Processed(message)`. :::note[Queue job trigger] Queue jobs have no schedule by default, so `EnqueueAsync()` is enough to start processing. Implement `DefaultSchedule` with a valid cron expression when the queue should instead be processed only on that schedule. Clearing or setting the schedule in the API or Admin UI switches between the two behaviors. ::: For batch workloads, configure `ActivationDelay` on the queue to collect a short burst before the first run starts: ```cs [Queue(ActivationDelay = "250ms")] public class ExampleQueueMessage : IQueueMessage { } ``` The activation delay is a fixed collection window. New messages do not keep extending it. It is different from `ProcessingDelay`, which controls when each message becomes eligible for processing. ## Filtering messages Use [generated queues and jobs](./generated.md) when a job should process only a filtered subset of a message type. A generated queue job can provide a configured `MessageFilter` expression. Job parameters do not currently derive or modify this filter. V3 removed the old filtered queue-job interfaces. ## Returning a custom status Sometimes it's not specific enough to say that a queue message was either processed or failed. You might have different actions you need to take for different outcomes that you want to specifically monitor. Nexus allows you to process a message and set it directly in a custom status rather than one of the built in statuses. In a batched queue job you call the `CustomStatus` method on the batch: ```cs var processingLog = "... something descriptive of what happened ..."; batch.CustomStatus(message, "MyCustomStatus", processingLog, isError: true); ``` And in a regular queue job you return `ProcessResults.CustomStatus()`: ```cs var processingLog = "... something descriptive of what happened ..."; return ProcessResults.CustomStatus("MyCustomStatus", processingLog, isError: true); ``` The `isError` argument lets you determine if the job should be marked as failed or not because of this custom status. Note that it's up to you to monitor queues for these custom statuses and take action. If you want to be notified about it you can implement a [custom healthcheck](../healthchecks/index.md) that checks if any queue has such a status. ## Batch size By default the batch size of a job is 100. You can control this by implementing the `BatchSize` property from `IBatchQueueJob`. Like this: ```cs public class MyBatchJob : IBatchQueueJob { public int BatchSize => 200; } ``` ## Number of messages per job run By default a Nexus queue job will run until the queue is empty. Sometimes that can be too aggressive. It might put too much pressure on other systems that the job is communicating with, and digging into the logs of that single run can be hard since it'll contain tons of logs. Nexus lets you set a max number of messages that a single job run can process by implementing the property `MaxNumberOfMessagesPerRun` in your queue job. Like this: ```cs public class MyQueueJob : IQueueJob // Or IBatchQueueJob { public int MaxNumberOfMessagesPerRun => 1000; } ``` ## Automatic progress reporting Queue jobs do not report progress automatically by default. This lets processing start with a lease immediately instead of first counting every pending message. You can opt into automatic progress by implementing `AutomaticProgressMode`: ```cs public class MyQueueJob : IQueueJob // Or IBatchQueueJob { public QueueJobProgressMode AutomaticProgressMode => QueueJobProgressMode.ProcessedCount; } ``` The available modes are: - `None` (the default) does not count pending messages or report automatic progress. - `ProcessedCount` reports how many messages have been handled, without counting the queue first. The Admin UI shows an indeterminate progress bar and the processed count. - `Percentage` counts pending messages before processing, then reports both the handled count and total. This preserves percentage-based progress at the cost of an additional queue query before every job run. Progress updates are aggregated per lease or batch instead of being written for every individual message. Code in the job can still inject and use [`IJobProgress`](../jobs/progress.md) in any mode, including `None`. For queues using external pending storage, the count used by `Percentage` can be approximate and may require an additional broker management operation or permission. The queue can also change after it is counted. Use `ProcessedCount` when avoiding that overhead is more important than showing a percentage. ## Multithreading The interfaces `IBatchQueueJob` and `IQueueJob` have a property called `NumberOfProcessingThreads` which by default returns `1`. If you want your job to process messages in multiple threads in parallel you can implement the method in your job to increase parallelism like this: ```cs public class ExampleQueueJob : IQueueJob { public int NumberOfProcessingThreads => 10; // Other methods and properties excluded for brevity } ``` Make sure not to store any instance state that isn't thread safe when you turn on multi threading. The same instance of your job will be used by all threads so if you need to store instance state make sure to use concurrent collections and/or locks. ## Controlling the job result message By default Nexus will create a job result message based on the number of messages processed by the job run. To create your own result message, implement `IQueueJobBase.GetResultsText()`: ```cs public class ExampleQueueJob : IQueueJob { public async Task ProcessMessageAsync(ExampleQueueMessage message, CancellationToken cancellationToken) => ProcessResults.Completed(); public string GetResultsText(QueueJobProcessResult result) { if (result.FailedCount == 0) { return null; // returning null means we'll get the default result message } return $"ERROR! {result.FailedCount} messages failed"; } } ``` ## Writing your own queue processing You don't have to use the built-in executor. A custom job can use `IQueueReader.LeaseAsync()`, `IQueueItemUpdater.ProcessedAsync()`, and `IQueueItemUpdater.FailedAsync()` directly. The job should implement both `IScheduledJob` (with its own `ExecuteAsync` and schedule) and `IQueueJobBase` so it has a custom execution entry point and the UI/API can identify its queue. Implementing only the marker base does not supply an executable job handler. ## Scheduling a processed message to be processed again Sometimes when you're using [idempotent messages](./index.md#idempotent-messages) to update external systems you might have dates in your message that affects the outcome of the processing. Such as a product message which contains a list of prices that have a valid from and to date. At the time of processing you pick the price that is valid right now, but you want to run the processing again when the next price becomes valid. For a batch processing job you can call: ```cs batch.Processed(message, processAgainAtUtc: DateTime.UtcNow.AddHours(1)); ``` And for a job that process messages one by one you do: ```cs return ProcessResults.Completed(processAgainAtUtc: DateTime.UtcNow.AddHours(1)); ``` This will mark the queue message as `Processed` together with the date for when it should be processed again. Note that it's not marked as `Sleeping` like unprocessed, future messages. This to be able to differentiate between messages that have been processed from ones that are just scheduled for processing. The message status is then changed to `Pending` again at the date you set, and the queue processing job is called again with the same message. If a change is enqueued for the same product/message before the date you set the status for the message becomes `Pending` directly and it's up to the job to recalculate when/if the message should get processed again. ## Named queues and processing settings One queue job implementation can process several [named queues](named.md), with independent job names, runs, parameters, and state. Use the `ProcessMessageAsync` overload with `QueueProcessingContext` or `batch.QueueName` to identify the queue. The Processing tab in the new Admin UI can override processing threads, lease time, and batch size per queue; active runs retain their snapshot and use updated settings on their next run. See [processing settings](processing-settings.md) for supported overrides and reset behavior. For example, a single-message handler can take the context explicitly: ```cs public Task ProcessMessageAsync( ExampleQueueMessage message, QueueProcessingContext context, CancellationToken cancellationToken) { var queueName = context.QueueName; return Task.FromResult(ProcessResults.Processed()); } ``` Existing handlers without the context argument continue to work. A batch belongs to one queue and exposes both `batch.QueueName` and `batch.Context`. --- # Processing directly from Kafka topics Nexus allows you to use Kafka as an external pending storage, which means that Nexus reads and processes messages directly from Kafka without first storing them in an SQL backed Nexus queue. Use it by first installing `CommerceMind.Nexus.Kafka` and then configuring the external storage: ```cs builder.Services .AddNexus() .AddKafkaExternalPendingStorage(options => { options.BootstrapServers = "localhost:9092"; options.ConsumerGroupId = "my-app-processing"; options.ReadinessConsumerGroupId = "my-app-readiness"; }); ``` And then on your queue: ```cs [Queue("export_order", StorageMode = QueueStorageMode.ExternalPending)] public class ExportOrderQueueMessage : IQueueMessageWithId { } ``` The processing consumer and readiness bridge read the same topic through different consumer groups. The processing group owns leasing and completion. The readiness group observes every record independently and commits its offset only after activation is durable. The two group IDs must be different. With the default prefix, queue `export_order` maps to topic `nexus-export_order`. Direct producers publish the Nexus external-pending envelope to that topic; no companion message is needed. Existing topics can be used after configuring the prefix and distinct groups, and the readiness group will catch up from its own committed offsets. Read more about [external pending storage here](./external-pending.md#readiness-bridges-and-direct-producers). # Enqueueing from Kafka topics Nexus has built-in support to use the very popular Apache Kafka platform for both [instance events](../general/instance-events.md) as well as taking messages from a Kafka topic and enqueueing into a Nexus queue. In order to use this you need to install the NuGet package called `CommerceMind.Nexus.Kafka`. You set it up like this: ```cs builder.Services .AddNexus() .AddQueues() .AddKafka(options => { options.ConsumerConfig.BootstrapServers = "..."; options.ConsumerConfig.GroupId = "..."; }) .AddNexusKafkaQueues(options => { // This allows you to control things like which status or when messages are processed options.EnqueueContextFactory = message => new EnqueueContext { Priority = message.Id != null ? 10 : 0 } }) .EnqueueFrom("kafka-topic"); ``` With this examples all messages posted to the `kafka-topic` will get placed in the Nexus queue for `KafkaQueueMessage`. ## Instance events You can also use Kafka as the way to [send signals](../general/instance-events.md) between Nexus instances in a scaled out environment. ```cs builder.Services .AddNexus() .AddQueues() .AddKafka(options => { options.ConsumerConfig.BootstrapServers = "..."; options.ConsumerConfig.GroupId = "..."; options.ProducerConfig.BootstrapServers = "..."; }) .AddKafkaInstanceEvents(options => { // Change this to something unique if you're using the same Kafka instance for multiple // different Nexus applications options.TopicName = "nexus_instance_events"; }); ``` --- # Queue maintenance Nexus runs queue maintenance independently from the scheduled-job loop. The job loop therefore does not query every queue on every iteration, and maintenance work cannot delay the scan that starts jobs. The maintenance worker has two paths: - **Readiness** promotes scheduled and retrying messages whose `ProcessAtUtc` has elapsed, and recovers expired leases. Nexus tracks retry and lease deadlines created by the current process and wakes at the nearest one. A periodic fallback scan handles messages created by another application instance, direct database changes, and deadlines restored after a restart. - **Cleanup** removes expired processed messages, history, and statistics on a separate cadence. Deletes are split into bounded batches to avoid long transactions and large lock spikes. The defaults suit most applications. Applications with many queues can reduce idle database work by increasing the fallback interval. Decreasing it reduces the worst-case delay for work created by another application instance. ```cs services.AddNexus() .AddQueues(options => { options.MaintenanceFallbackInterval = TimeSpan.FromSeconds(30); options.CleanupInterval = TimeSpan.FromMinutes(5); options.CleanupBatchSize = 2_000; options.MaxCleanupBatchesPerRun = 5; }); ``` `CleanupBatchSize * MaxCleanupBatchesPerRun` is the maximum number of rows that one cleanup pass deletes from each retention store for a queue. If more expired rows remain, later cleanup passes continue the work. `IQueueItemUpdater.OrganizeMessagesAsync()` remains available for applications that invoke maintenance explicitly. It now uses the same split readiness and bounded-cleanup implementation; passing `force: true` bypasses its compatibility throttle. --- # Named queues Named queues let several independently processed queues share a CLR message type. Define the destinations when registering queues: ```cs public class LanguageMessage : IQueueMessageWithId { public string? Id { get; set; } public string? Text { get; set; } } builder.Services.AddNexus().AddQueues(options => { options.NamedQueues = [ new NamedQueue { Name = "language_sv", DisplayName = "Swedish" }, new NamedQueue { Name = "language_en", DisplayName = "English" } ]; }); ``` Queue names identify persistent storage and must remain stable. Use letters, digits, and underscores. Virtual queues have no storage, so their names may also contain hyphens and periods. Nexus rejects names that collide with another queue's backing tables, including provider identifier truncation. Definitions support the usual per-queue retention, retry, health, category, storage-mode, external-provider, processing-delay, and activation-delay options. Explicit named definitions suppress automatic concrete queue discovery for their message type. Queue names are unique, ignoring case, across concrete and virtual queues. ## Contributions from libraries Libraries can call `AddNexus().AddQueues()` and `AddJobs()` independently, in either library order. Each method shares its options across calls. Append to `NamedQueues`, `GeneratedQueues`, `GeneratedJobs`, and assembly lists with `Add` or `AddRange`; assigning a new list replaces earlier definitions. Scalar settings use the last configured value, including job background-worker settings. ```cs var nexus = builder.Services.AddNexus(); // Library A nexus.AddQueues(options => options.NamedQueues.Add( new NamedQueue { Name = "language_sv", DisplayName = "Swedish" })); nexus.AddJobs(options => options.AssembliesToScanForJobs.Add(typeof(LanguageJob).Assembly)); // Library B may contribute more queues after Library A registered its jobs. nexus.AddQueues(options => options.NamedQueues.Add( new NamedQueue { Name = "language_en", DisplayName = "English" })); nexus.Build(); ``` Call `Build()` once after all contributions. Nexus registers the complete queue set before binding jobs and distributed processors, so an ordinary `LanguageJob` discovered by Library A also binds to Library B's queue. Each assembly scan uses the filter configured at that call; discovering a type again does not register it twice. Explicit named definitions suppress ordinary concrete discovery even when contributed later. Repeated calls without new definitions do not duplicate registrations. Adding duplicate queue or job names fails at `Build()`, ignoring case, as do conflicting processing implementations. Generated job bindings still require their concrete queues to be declared before the contributing `AddJobs()` call. An explicit generated binding takes precedence over automatic binding of the same job implementation to that queue. Registering queues alone does not enable processing: call `AddJobs()` or `AddQueueProcessors()` explicitly. Call `AddQueues()` before `AddQueueProcessors()`. ## Enqueueing Supply the queue name on every dispatch to a named queue: ```cs await enqueuer.EnqueueAsync("language_sv", new LanguageMessage { Id = "123", Text = "Hej" }); await enqueuer.EnqueueAsync("language_en", messages); ``` This applies even when there is only one named queue for the type. `EnqueueAsync(message)` cannot infer a named destination. Adding or removing a language must not change whether a type-only call is valid. An unknown name, incompatible payload, or ambiguous unnamed destination fails with an error. For a queue-bound typed accessor, explicitly select the name through `IQueueResolver`: ```cs var swedish = queues.GetEnqueuer("language_sv"); await swedish.EnqueueAsync(message); var reader = queues.GetReader("language_sv"); var updater = queues.GetUpdater("language_sv"); ``` Once selected, the accessor's ordinary methods operate on that queue. Advanced custom processing can also enumerate `IQueueReader` or `IQueueItemUpdater` and select by `QueueName`. Unqualified typed DI resolution cannot choose among named destinations. Validation and priority calculation remain type-based. The validator for `LanguageMessage` applies to every concrete queue receiving that type, unless `SkipValidation` is set. Explicit enqueue priority takes precedence over `IMessagePriorityCalculator`. Applications can include routing information on their own payload if these hooks need it. Message identity, idempotency, previous successfully processed versions, history, partial merges, and processing updates are local to each queue. The same message ID can exist independently in both queues. ## Processing One ordinary processor class handles every concrete queue registered for its message type: ```cs public class LanguageProcessor : IQueueProcessor { public Task ProcessMessageAsync( LanguageMessage message, QueueProcessingContext context, CancellationToken cancellationToken) { var queueName = context.QueueName; return Task.FromResult(ProcessResults.Processed()); } } ``` Queue jobs work the same way. A job binding has its own name, schedule, pause state, and history for each queue; named bindings use `{job-name}_{queue-name}` by default. Existing single ordinary queue jobs keep their names. Explicit generated job definitions can preserve existing per-queue job names and metadata. Execution is independent per queue. With four workers per instance and ten queues, each instance can run forty workers. Each queue has its own leases, retries, activation, and administrative state. A queue can have one queue job or one distributed processor. Existing storage-mode restrictions still apply. Single-message jobs and processors can receive `QueueProcessingContext`. Existing implementations of the overload without context continue to work. Batches expose `batch.QueueName` and `batch.Context`; each batch belongs to one queue. Named queues do not require generic subclasses of the message or job implementation. ## Virtual entry points Keep the ordinary different-type transformer for the common case: ```cs public TransformResult Transform(OrderPlacedMessage message, EnqueueContext? context) => new(new ExportOrderMessage { Id = message.Id }); ``` Select a named destination with one extra argument: ```cs return new(message, queueName: "language_sv"); ``` Or send to several named queues through the same result type: ```cs return TransformResult.ToMany([ new("language_sv", message), new("language_en", message) ]); ``` Register an explicit virtual queue using `NamedQueue` with `IsVirtual = true` and `TransformerTypes = [typeof(MyTransformer)]`. Its input may use the same CLR type as its concrete destinations. An ordinary `[VirtualQueue]` registration remains supported. The virtual queue has its own name and no storage or processing settings. Incoming enqueue context is inherited unless a result/destination replaces it. Empty results filter out a message. Forwarding to another virtual queue is supported; routing cycles fail with a path description. Enqueue into the virtual queue's name to request routing, or into a concrete name to choose a destination directly. Multiple destinations do not imply an atomic transaction across providers. A retry after partial delivery may repeat destinations that already succeeded. ## Runtime processing settings The Processing tab in the new Admin UI lets you override the code defaults for one concrete queue. Distributed processors expose workers per instance, lease size, lease time, and batch size for batch processors. Queue jobs expose processing threads, lease time, and batch size for batch jobs. Empty fields restore the implementation's defaults; **Use code defaults** resets all overrides. Overrides are stored in individual nullable columns on the queue metadata table and propagate to all instances. A null column uses the implementation's code default. Active queue jobs finish with their existing settings and use new values on the next run. Distributed processors finish their current work and rebuild their worker pools before leasing more work with new settings. A busy queue does not need to become empty. Lease-time changes affect future leases; existing lease expiry times remain unchanged. The API exposes `GET` and `POST /api/queues/{queueName}/processing-settings`. Reads include defaults, saved overrides, effective values, and whether running processor instances still report older settings. POST replaces all saved overrides; omitted or null fields use code defaults, and posting `{}` resets every override. Counts and lease durations must be positive. Custom processing loops can read metadata explicitly; these controls apply automatically to built-in executors. The legacy Admin UI retains its existing functionality and does not expose these new controls. See [processing settings](processing-settings.md) for the full override API and reset semantics. ## Separate producers and broker ingress An API-only producer can share the message CLR contract and queue names with the Nexus host and choose a name dynamically through `IApiEnqueuer`: ```cs await apiEnqueuer.EnqueueAsync("language_" + language, message); ``` Register API enqueuers for the contract assembly as usual. No per-name registration, local queue storage, processors, or local virtual transformers are required. The host validates the named destination. Enqueueing to a remote virtual queue runs its transformers on the host. Azure Service Bus, RabbitMQ, and Kafka ingress registrations accept an optional `nexusQueueName` argument. It selects a concrete destination or a virtual entry point; message-dependent routing belongs in the virtual transformer. Existing ordinary type-based ingress remains supported. External-pending storage continues to isolate broker resources by the concrete queue name. --- # Message priority The default behavior in Nexus is to process messages in the order that they come in; oldest first. In some cases this is not granular enough and you have some messages that needs to be prioritized over others. Nexus allows you to control this by passing a number as priority when you enqueue the message. Eg: ```cs public class MyController(IEnqueuer enqueuer) { public async Task Enqueue() { await enqueuer.EnqueueAsync(new MyMessage { ... }, new EnqueueContext { Priority = 10 }); return Ok(); } } ``` Database and in-memory queues lease eligible messages by highest priority first, then oldest internal ID. This controls lease selection, not completion order: already leased messages keep processing, and parallel workers or instances can finish in a different order. External pending storage follows its provider's priority and ordering support. Use a single processing thread when the application needs sequential handling, and account for retries and delayed messages. ## Default priority The default priority in Nexus is `0`. When you enqueue messages you're encouraged to create an enum or a static class with your named priority levels rather than setting numbers like in the example above. Note that you can have low prio messages that have a negative priority. Which means that they will be processed after any message with the default (`0`) priority. ## Programmatically determining the priority In some cases messages comes in through the API and you don't want to let the external party dictate the priority, but if they use the Nexus API directly to enqueue you don't get to set the priority. For this scenario and others you can implement `IMessagePriorityCalculator` and register with `IServiceCollection`. ```cs public static class MessagePriority { public const int High = 100; public const int Low = 0; } public class MyMessagePriorityCalculator : IMessagePriorityCalculator { public async Task CalculatePriorityAsync(MyMessage message) { if (message.SomeProperty == "some value") { return MessagePriority.High; } return MessagePriority.Low; } } ``` Note that this class is only called for messages that doesn't have an explicit priority set already. --- # Queue processing settings Processing implementations supply code defaults. The new Admin UI's Processing tab can save overrides for an individual concrete queue without redeploying the application. Overrides are shared across instances and survive restarts. ## Settings for each processing model | Setting | Queue job | Distributed processor | | --- | --- | --- | | `NumberOfProcessingThreads` | Threads within the single active run | Not applicable | | `ConcurrencyPerInstance` | Not applicable | Workers on every instance | | `LeaseSize` | Not exposed as a runtime override | Messages per leased chunk | | `BatchSize` | Batch jobs only | Batch processors only | | `MessageLeaseTime` | Duration for future leases | Duration for future leases | Counts and lease time must be positive. A non-batch implementation cannot accept a batch-size override, and settings for the other processing model are rejected. Virtual queues have no processing settings of their own. Distributed concurrency multiplies across queues and hosts. Four workers on each of three instances can process up to twelve chunks for one queue. Ten such named queues can have ten independent pools. Size this against the capacity of downstream systems and the database. Choose a lease time longer than the worst expected processing time of a whole leased chunk or batch. Lease expiry can permit another worker to process the message while the original handler is still running. More workers or a larger batch does not renew existing leases automatically. ## Code defaults and saved overrides Each setting has a nullable saved override. Null uses the implementation's current code default. Clearing a field in the Processing tab restores that default; **Use code defaults** clears all overrides. A code change only changes an effective setting when no saved override replaces it. Active queue jobs keep the settings captured at the start of the run; new settings apply on the next run. Distributed processors finish current work and rebuild their pools before leasing more work with changed settings. The queue does not need to become empty first. New lease times apply to future leases; existing lease expiry dates stay unchanged. The legacy Admin UI does not expose these controls. Schedules, activation/processing delay, pause state, maximum messages per job run, retry policy, and automatic progress are separate settings, not fields in this override record. ## HTTP API Routes are relative to `NexusApiOptions.BasePath`: ```text GET /api/queues/{queueName}/processing-settings POST /api/queues/{queueName}/processing-settings ``` Reads include `kind` (`Job` or `Distributed`), `processesBatches`, `defaults`, `overrides`, `effective`, and `pendingChanges`. For distributed processors, `pendingChanges` indicates running instances still reporting older active settings. It is not an acknowledgement that a running queue job has adopted new values. For example, set distributed worker count and lease time: ```json { "concurrencyPerInstance": 4, "messageLeaseTime": "00:10:00" } ``` The POST replaces the saved override record. Omitted/null fields revert to code defaults; it is not a partial patch preserving other overrides. Include every override that should remain. Posting `{}` clears them all. Application code can use the scoped `IQueueMetaDataRepository`: ```cs await queueMetaDataRepository.SetProcessingSettingsAsync("order_exports", new QueueProcessingSettings { NumberOfProcessingThreads = 4, MessageLeaseTime = TimeSpan.FromMinutes(10) }); ``` This example applies to a queue job. `GetProcessingSettingsAsync()` returns the saved overrides, not resolved code defaults. Built-in executors apply overrides automatically; custom processing loops must read and apply the relevant settings themselves. Custom metadata repositories must implement persistence if they support these controls. ## Processor status and pause controls Distributed processors appear on the queue page rather than in the jobs list. The API provides: ```text GET /api/queues/{queueName}/processor POST /api/queues/{queueName}/processor/pause POST /api/queues/{queueName}/processor/resume POST /api/queues/{queueName}/processor/wake ``` Status includes workers, in-flight messages, recent errors, and instance reports. Pause is durable across the cluster: workers finish or release current work and stop taking leases. Resume requests readiness again. Wake requests a readiness check and does not bypass pause or message eligibility. Queue jobs use job enable/disable and start controls instead. --- # Distributed queue processors Distributed queue processors are an opt-in alternative to queue jobs for high-throughput queues. A queue job has one active run across the Nexus cluster. A distributed processor runs a worker pool on every application instance and relies on Nexus's atomic message leases to divide the work safely. Use a distributed processor when messages can be processed independently and adding application instances should increase throughput. Continue to use a queue job when you need job runs, schedules, parameters, mutexes, progress, or strict single-run behavior. ## Registration Register queues first and jobs before processors if the application uses both: ```csharp services .AddNexus() .AddQueues() .AddJobs() .AddQueueProcessors() .AddSqlServerConnection() .Build(); ``` `AddQueueProcessors()` scans the entry assembly. Additional assemblies and filtering can be configured with `QueueProcessorOptions`. Processor bindings are created at `Build()` after all queue and job contributions. A discovered processor also binds to matching concrete queues contributed by later libraries. Repeated calls share options and do not duplicate bindings. See [library contributions](named.md#contributions-from-libraries). Each concrete queue can have either one queue job or one distributed processor. Nexus rejects conflicting registrations during startup. In-memory queues are intentionally not supported because their leases cannot be shared across application instances. Database and external-pending queues use their existing atomic lease implementation. ## Processing one message at a time ```csharp [Queue("order_updates")] public class OrderUpdate : IQueueMessage { public string OrderId { get; set; } = ""; } public class OrderUpdateProcessor : IQueueProcessor { public int ConcurrencyPerInstance => 8; public TimeSpan MessageLeaseTime => TimeSpan.FromMinutes(5); public async Task ProcessMessageAsync( OrderUpdate message, CancellationToken cancellationToken) { await ApplyUpdateAsync(message, cancellationToken); return ProcessResults.Processed(); } } ``` `ConcurrencyPerInstance` is multiplied by the number of running application instances. For example, eight workers on three instances allows up to 24 leased chunks to make progress concurrently. A processor instance is created in a fresh dependency-injection scope for each leased chunk. That instance is called sequentially and is never invoked concurrently. Single-message processors lease one message per worker by default. Override `LeaseSize` to opt into larger chunks. Batch processors keep a default lease size of 50, capped by `BatchSize`. ## Batch processing ```csharp public class SearchIndexProcessor : IBatchQueueProcessor { public int ConcurrencyPerInstance => 2; public int BatchSize => 100; public async Task ProcessMessagesAsync( QueueMessageBatch batch, CancellationToken cancellationToken) { await IndexAsync(batch.Messages, cancellationToken); batch.Processed(batch.Messages); } } ``` Batch processors use the same result methods as batch queue jobs. Messages without an explicit result are released back to `Pending`. ## Delivery and ordering Distributed processing is at-least-once. A message can be processed again after a worker crash or an expired lease, so handlers should be idempotent where possible. No ordering is guaranteed across workers or instances. Set `MessageLeaseTime` longer than the longest expected processing time for one leased chunk or batch. A lease that expires while processing can allow another instance to receive the same message. ## Administration The new Admin UI shows a distributed processor on its queue rather than in the jobs list. Its Processing tab includes configuration, aggregate cluster activity, and the recent state of each instance. The queue page can pause, resume, or wake the processor. Pausing is cluster-wide and durable. Workers finish or release their current work and stop taking new leases. Resuming wakes every instance through the normal durable queue-activation mechanism. ## Named queues and runtime settings One processor class can handle several [named queues](named.md), with independent execution per queue. Use the `ProcessMessageAsync` overload accepting `QueueProcessingContext` to read `context.QueueName`; batches expose `batch.QueueName`. Worker, lease, and batch settings can be overridden per queue through metadata and the Processing tab in the new Admin UI. Changes finish existing work before replacing the worker pool and apply across all instances. See [processing settings](processing-settings.md) for the supported fields, replacement/reset behavior, and API controls. --- # Processing directly from RabbitMQ queues Nexus allows you to use RabbitMQ as an external pending storage, which means that Nexus reads and processes messages directly from RabbitMQ without first storing them in an SQL backed Nexus queue. Use it by first installing `CommerceMind.Nexus.RabbitMQ` and then configuring the external storage: ```cs builder.Services .AddNexus() .AddRabbitMqExternalPendingStorage(options => { options.ConnectionString = new Uri("amqp://guest:guest@localhost/"); }); ``` And then on your queue: ```cs [Queue("export_order", StorageMode = QueueStorageMode.ExternalPending)] public class ExportOrderQueueMessage : IQueueMessageWithId { } ``` For each Nexus queue, v3 declares a durable fan-out exchange and two durable queues. With the defaults, `export_order` uses: - Exchange `nexus:external-pending:export_order`. - Processing queue `nexus:export_order`. - Readiness queue `nexus:export_order:readiness`. Nexus leases messages only from the processing queue. The readiness bridge consumes the readiness queue and acknowledges a notification only after activation is durable. Direct producers must publish to the exchange, not directly to the processing queue, so both paths receive the message. Existing producers that publish directly to `nexus:export_order` must be migrated before the queue job is changed to readiness-only mode. Read more about [external pending storage here](./external-pending.md#readiness-bridges-and-direct-producers). # Enqueueing from RabbitMQ queues RabbitMQ is a great way to communicate between different applications. But sometimes you want to have more visibility into queue messages, when they're processed and if they fail. And to be able to retry in a controlled fashion when they fail. Nexus lets you get the best of both worlds by automatically enqueueing from a RabbitMQ queue to a Nexus queue like the example below. In order to use this you need to install the NuGet package called `CommerceMind.Nexus.RabbitMQ`. ```cs builder.Services .AddNexus() .AddQueues() .AddRabbitMq(options => { options.UserName = "..."; options.Password = "..."; options.HostNames = ["localhost"]; // Or: options.ConnectionString = new Uri("amqp://..."); }) .AddNexusRabbitMqQueues(options => { // This allows you to control things like which status or when messages are processed options.EnqueueContextFactory = message => new EnqueueContext { Priority = message is MyImportantQueueMessage ? 10 : 0 } }) .EnqueueFrom(new NexusRabbitMqQueue { Name = "rabbit_queue", Bindings = [new NexusRabbitMqQueueBinding { ExchangeName = "my_exchange", RoutingKey = "" }] }); ``` ## Instance events You can also use RabbitMQ to [send signals](../general/instance-events.md) between Nexus instances in a scaled out environment. ```cs builder.Services .AddNexus() .AddQueues() .AddRabbitMq(options => { options.UserName = "..."; options.Password = "..."; options.HostNames = ["localhost"]; // Or: options.ConnectionString = new Uri("amqp://..."); }) .AddRabbitMqInstanceEvents(options => { // Change this to something unique if you're using the same RabbitMQ instance for multiple // different Nexus applications options.ExchangeName = "nexus_instance_events"; }); ``` --- # Processing directly from Redis Nexus allows you to use Redis as an external pending storage, which means that Nexus reads and processes messages directly from Redis without first storing them in an SQL backed Nexus queue. Use it by first installing `CommerceMind.Nexus.Redis` and then configuring the external storage: ```cs builder.Services .AddNexus() .AddRedisExternalPendingStorage(options => { options.ConnectionString = "localhost:6379"; }); ``` And then on your queue: ```cs [Queue("export_order", StorageMode = QueueStorageMode.ExternalPending)] public class ExportOrderQueueMessage : IQueueMessageWithId { } ``` For a queue named `export_order`, the default topology uses keys with the same Redis Cluster hash tag: - `nexus:queue:{export_order}` is the ready-message sorted set. - `nexus:queue:{export_order}:delayed` is the delayed-message sorted set. - `nexus:queue:{export_order}:readiness` is the durable readiness stream. - `nexus:queue:{export_order}:idempotent` stores idempotency markers. Nexus enqueueing writes the message and readiness stream entry atomically. A direct producer must do the same with a Lua script; publishing only to the sorted set is not enough for a readiness-only queue job. The readiness consumer group is independent of message processing and acknowledges a stream entry only after queue activation is durable. See [external pending storage](./external-pending.md#readiness-bridges-and-direct-producers) for the envelope and script contract. Redis can also hold Nexus's coalesced activation state instead of SQL: ```cs builder.Services .AddNexus() .AddRedisQueueActivations(options => { options.ConnectionString = "localhost:6379"; options.KeyPrefix = "nexus"; }); ``` This is separate from Redis external pending storage: either feature can be used without the other. When upgrading existing external-pending queues, drain or explicitly migrate the old unbraced Redis keys. V3 does not read them automatically. Include delayed messages and any identity/idempotency state in the cutover. Read more about [external pending storage here](./external-pending.md). # Enqueueing from Kafka topics Nexus has built-in support to use the very popular Apache Kafka platform for both [instance events](../general/instance-events.md) as well as taking messages from a Kafka topic and enqueueing into a Nexus queue. In order to use this you need to install the NuGet package for RabbitMQ called `CommerceMind.Nexus.Kafka`. You set it up like this: ```cs builder.Services .AddNexus() .AddQueues() .AddKafka(options => { options.ConsumerConfig.BootstrapServers = "..."; options.ConsumerConfig.GroupId = "..."; }) .AddNexusKafkaQueues(options => { // This allows you to control things like which status or when messages are processed options.EnqueueContextFactory = message => new EnqueueContext { Priority = message.Id != null ? 10 : 0 } }) .EnqueueFrom("kafka-topic"); ``` With this examples all messages posted to the `kafka-topic` will get placed in the Nexus queue for `KafkaQueueMessage`. ## Instance events You can also use Kafka as the way to [send signals](../general/instance-events.md) between Nexus instances in a scaled out environment. ```cs builder.Services .AddNexus() .AddQueues() .AddKafka(options => { options.ConsumerConfig.BootstrapServers = "..."; options.ConsumerConfig.GroupId = "..."; options.ProducerConfig.BootstrapServers = "..."; }) .AddKafkaInstanceEvents(options => { // Change this to something unique if you're using the same Kafka instance for multiple // different Nexus applications options.TopicName = "nexus_instance_events"; }); ``` --- # Storing processed messages The default behavior of a Nexus queue is to delete messages when they have been processed. But for some use cases it's interesting to keep the processed messages. You can control how for long processed messages are stored on the `[Queue]` attribute like this: ```cs [Queue("example", DisplayName = "Example queue", ProcessedMessagesRetention = "10")] public class ExampleQueueMessage : IQueueMessage { } ``` In this example the messages are stored for 10 days. The property `ProcessedMessagesRetention` should be either a string that can be parsed into a `TimeSpan` by [`TimeSpan.Parse()`](https://docs.microsoft.com/en-us/dotnet/api/system.timespan.parse?view=net-6.0) or the special string `"forever"` which tells Nexus to never delete processed messages. The attribute property was renamed from `KeepProcessedMessagesFor` to `ProcessedMessagesRetention` in v3. The global default is `QueueSystemOptions.ProcessedMessagesRetention`, and runtime per-queue overrides are available through the Admin UI/API or `IQueueMetaDataRepository.SetProcessedMessageRetentionAsync()`. Cleanup runs in [bounded maintenance batches](maintenance.md), so expired rows may remain until a cleanup pass. ## Use cases One use case for storing processed messages is to enable troubleshooting of important messages. Let's say that you have a queue of orders that should get exported to another system. The export from your side might have worked perfectly but the receiving system might reach out to you to say "hey could you send the message for order 123 again?" or "hmm what exactly did you send to us for order 123?". If you delete processed messages you'd have to dig through logs to maybe be able to reconstruct the message. If you instead keep processed messages you can easily answer both questions. Another use case is when an external system is only able to send full exports of data rather than incremental changes. The system might be scheduled to export everything daily into a Nexus queue. If you store processed messages and turn on [idempotent messages](./index.md#idempotent-messages) on your queue Nexus will only set messages to `Pending` where the data differs from yesterday. A third use case is to do change tracking inside your queue job by reading the [previous version](./index.md#previous-message-version) of the message and compare the current to the previous to see what has changed. For ordinary completed/deleted items, retaining the message keeps this state available for the next enqueue. If a newer version arrives while the old one is leased, the `AwaitingRelease` path preserves the successfully processed version as previous data even when processed-message retention is zero. See [message identity](identity.md#enqueueing-while-a-message-is-leased). --- # Retrying queue messages When you implement a queue processing job you might end up in a situation where you can't process the message right now but you know that you'll soon want to try it again. Such as if you detect that a system that you want to send the message to is down at the moment so you want to try again in X minutes. Nexus lets you handle that by returning `ProcessResults.RetryLater()` from a queue job or call `batch.RetryLater()` in a batch queue job. You specify the amount of time to wait and then Nexus will handle the rest. Nexus will place the message in a status called `AwaitingRetry` and it will sit there until it is time to try again. At that point queue maintenance sets the status to `Pending` and requests a durable queue activation, so a readiness-driven queue job starts without needing a cron schedule. ## Automatic retries You can also instruct Nexus to automatically retry messages in a queue. In the Admin UI you can click on the More button on the queue page and set the field "Automatic error retry" to eg `24h` (or `1d` or `1h 30m` etc) which will automatically retry any message that gets the Error status after that time period. Note that the time period is per message, and does not run periodically to batch update all Error messages to Pending. You can also control this through code by either setting it in the `[Queue]` attribute: ```cs [Queue("my_queue", AutomaticRetryAfter = "00:15:00")] public class MyQueueMessage : IQueueMessage { } ``` `AutomaticRetryAfter` represents a string that can be passed to `TimeSpan.Parse()` so in the example above it'll be retried 15 minutes after it first failed. Another way of setting it through code is to use `IQueueMetaDataRepository.SetRetryErrorsIntervalAsync()` which lets you set a runtime value for it. This is the same method that gets called when updating it in the Admin UI/API. If no value has been set through `IQueueMetaDataRepository` Nexus falls back to what's set in the attribute, and if no value is set in the attribute no automatic retries are performed. Automatic retries respects the max retries for a queue which is described below. ## Max retries To avoid having messages that end up in an inifinite loop of retries Nexus has a default limit on how many times a message can be retried. The default limit is 10 and once a message has reached that number of retries it will get the status `Error` and a comment saying `Message was retried too many times (10) and has been placed in the Error status`. At that point it will not be automatically retried again but you can always manually set it back to `Pending` which gives the message 10 more tries. If you want to change the limit of max number of retries you can either set it on all queues like this in your `Program.cs`: ```cs builder.Services.AddNexus().AddQueues(options => { options.DefaultMaxRetries = 5; }); ``` You can also set this per queue using the `[Queue]` attribute: ```cs [Queue("example", MaxRetries = 5)] public class ExampleQueueMessage : IQueueMessage { } ``` Note that you can allow an infinite number of retries by setting `-1` as the max. ## Status when max retries By default the status becomes `Error` when a message has been retried too many times. But you can control which status it gets by passing the status as part of the `ProcessResults.RetryLater()` call, like this: ```cs return ProcessResults.RetryLater(TimeSpan.FromMinutes(5), statusWhenMaxRetried: "MyErrorStatus"); ``` --- # Queue statistics Nexus can store statitics for your queues to help visualize the activity in a queue. If you've enabled statistics you can find them in the Admin UI under Statistics and in the queue list and queue details. The statistics shown in the queues list is the total number of status changes/total activity in a queue. The statitics consists of counters for each status. The statistics represents the changes in status rather than the current count of a status at any given time. That is, if the statistics reports zero Error messages it means that no messages got the status Error at that point in time. By default statistics are turned off but you can opt-in to them by either: **Turning it on for all queues by doing:** ```cs builder.Services.AddNexus().AddQueues(options => { options.DefaultStatisticsRetention = TimeSpan.FromDays(30); // TimeSpan.MaxValue to store forever }); ``` **Turning it on on a specific queue**: ```cs [Queue(StatisticsRetention = "30d")] public class MyQueueMessage : IQueueMessage { } ``` The string set for `StatisticsRetention` must either by a parsable `TimeSpan` or in the human readable format, eg `6h` or `30d`. **Updating through the Admin UI/API**: In the Admin UI you can go to the queue details and click on the More button and edit the Statistics retention per queue. The Admin UI uses the API to do this, so it's also possible to control through the API directly. See the Swagger docs for endpoint details. ## Disabling statistics By default statistics tables are created per queue to make it possible to enable it dynamically through the Admin UI. If you really don't want to see the statistics feature you can disable it completely by doing: ```cs builder.Services.AddNexus().AddQueues(options => { options.DisableStatistics = true; }); ``` This deletes all statistics tables and prevents statistics features from being displayed in the UI. ## `IQueueStatisticsReader` / `IQueueStatisticsWriter` The reader and writer work with `IDatabaseQueueStorage` and are scoped Nexus dependencies. They are interfaces you can override to either implement your own statistics storage or decorate to send the statistics to analytics. The writer interface gets called directly when any message status changes (or new messages are created), so you should store the changes in memory and persist the changes in batches in a background thread. --- # Queue message validation If you need to validate messages before they're added to Nexus there's two approaches. Either you can let your message class implement `IQueueMessageWithValidation` like this: ```cs public class MyMessage : IQueueMessageWithValidation { public required int MyProperty { get; set; } public ValidationResults Validate() { return MyProperty < 10 ? ValidationResults.Valid() : ValidationResults.Invalid("Value too high"); } } ``` Or you can implement a separate validator class like this: ```cs public class MyMessage : IQueueMessage { public required int MyProperty { get; set; } } public class MyMessageValidator : IQueueMessageValidator { public async Task ValidateAsync(MyMessage message) { return message.MyProperty < 10 ? ValidationResults.Valid() : ValidationResults.Invalid("Value too high"); } } ``` If you have a validator class you don't need to register it, Nexus will find it as long as it's placed in an assembly that's scanned for queue messages. If you need to place the validator in an assembly that's not scanned for messages you can explicitly add the validator by calling `AddQueueValidator()` on the Nexus registration. The effect of returning `ValidationResults.Invalid()` will be that a `QueueValidationException` will be thrown and in the case of the API the request will fail with a `400 Bad Request` and with the validation message as the response. ## Skipping validation In some cases you might want to explicitly skip validation, like if the message comes from an external source and it's more important to store the message than validating it. In such cases you can use `SkipValidation` on the `EnqueueContext` like this: ```cs await enqueuer.EnqueueAsync(message, new EnqueueContext { SkipValidation = true }); ``` ## API validation Note that some validation can happen even before your validators are called. Like if you have `required` properties on your message class and incoming API requests are missing such values. In that case the deserialization will fail and the API will respond with `400 Bad Request` before your validator is called. If you want to handle these cases yourself you either need to skip the `required` part of your properties or prevent the serialization from throwing by implementing your own `IQueueMessageSerializer`. --- # Virtual queues Depending on your application you might not want the message producer to be in complete control of the message that is added to a queue. And in some cases a single event should cause multiple messages to be added to different queues. But you don't want to force the message producer to have to think about that. To solve this the system has a concept of virtual queues. A virtual queue is an `IQueueMessage` that doesn't have it's own backing storage, it just acts as a proxy in front of one or more concrete queues. You can then register one or more message transformers to place messages in concrete queues. Hopefully this all makes more sense with a concrete example. Let's say we're in an e-commerce context and we receive an event that an order has been placed. When an order is placed we want to do at least two things; send an order confirmation email and export the order to some other system such as the ERP. The naive implementation of this would something like this: ```cs [Queue("send_order_confirmation")] public class SendOrderConfirmationQueueMessage : IQueueMessageWithId { public string? Id { get; set; } } [Queue("export_order")] public class ExportOrderQueueMessage : IQueueMessageWithId { public string? Id { get; set; } } public class OrderController { private readonly IEnqueuer _sendOrderConfirmationEnqueuer; private readonly IEnqueuer _exportOrderQueueMessageEnqueuer; public OrderController(IEnqueuer sendOrderConfirmationEnqueuer, IEnqueuer exportOrderQueueMessageEnqueuer) { _sendOrderConfirmationEnqueuer = sendOrderConfirmationEnqueuer; _exportOrderQueueMessageEnqueuer = exportOrderQueueMessageEnqueuer; } [HttpPost] public async Task OrderPlacedAsync(string orderId, string email) { await _sendOrderConfirmationEnqueuer.EnqueueAsync(new SendOrderConfirmationQueueMessage { Id = orderId, }); await _exportOrderQueueMessageEnqueuer.EnqueueAsync(new ExportOrderQueueMessage { Id = orderId, }); return NoContent(); } } ``` There's a couple of problems with this approach. The first one is that the controller needs to know which queues are interested in getting notified when orders are placed. This means that if we add an additional queue for order processing we'd need to update the controller to enqueue that message as well. The second problem is that the controller here is in charge of creating and setting properties to the message that is actually placed in the queue. In the above example it's really simple but constructing a message can be more complex than that. In simple applications this isn't really an issue but in larger application it becomes a problem. So let's update the example to use virtual queues instead! ```cs [VirtualQueue("order_placed")] public class OrderPlacedQueueMessage : IQueueMessageWithId { public string? Id { get; set;} } [Queue("send_order_confirmation")] public class SendOrderConfirmationQueueMessage : IQueueMessageWithId { public string? Id { get; set;} } public class SendOrderConfirmationQueueMessageSyncTransformer : IVirtualQueueMessageSyncTransformer { public TransformResult Transform(OrderPlacedQueueMessage message, EnqueueContext? context) { return new(new SendOrderConfirmationQueueMessage { Id = message.Id }); } } [Queue("export_order")] public class ExportOrderQueueMessage : IQueueMessageWithId { public string? Id { get; set;} } public class ExportOrderQueueMessageSyncTransformer : IVirtualQueueMessageSyncTransformer { public TransformResult Transform(OrderPlacedQueueMessage message, EnqueueContext? context) { return new(new ExportOrderQueueMessage { Id = message.Id }); } } public class OrderController { private readonly IEnqueuer _orderPlacedEnqueuer; public OrderController(IEnqueuer orderPlacedEnqueuer) { _orderPlacedEnqueuer = orderPlacedEnqueuer; } [HttpPost] public async Task OrderPlacedAsync(string orderId, string email) { await _orderPlacedEnqueuer.EnqueueAsync(new OrderPlacedQueueMessage { Id = orderId, }); return NoContent(); } } ``` So there's a lot of stuff to unpack here. First we have a new attribute called `[VirtualQueue]`. This is what tells the system that this message should not get a concrete queue storage setup, it's only used for enqueuing messages to other queues. The second thing is that we have two message transformers. These are just classes that implements either `IVirtualQueueMessageSyncTransformer` or it's async counterpart `IVirtualQueueMessageTransformer`. They're responsible for translating a virtual message into a concrete message. The message transformers are automatically registered with the service collection as long as they exists in assemblies which are scanned for `IQueueMessage` implementations. In the controller we now take in an `IEnqueuer` for our virtual queue. Regular queues gets more services registered such as `IQueueReader` and `IQueueItemUpdater` but the only thing you can do with a virtual queue is to add messages to it. What happens behind the scenes when you call the enqueuer is that the message transformers we created are called and the enqueuers for `SendOrderConfirmationQueueMessage` and `ExportOrderQueueMessage` are called behind the scenes. Return a `TransformResult` with a null transformed message to filter out the input, for example `return new(null);`. If every transformer returns an empty result, nothing is enqueued. The incoming enqueue context is inherited unless a result supplies a replacement. ### EnqueueContext If the call site that wants to enqueue messages includes an `EnqueueContext` it gets passed into your transformer as in the example above. But you can override the context by passing a new `EnqueueContext` to the `TransformResult` as the second argument, eg: ```cs public class ExportOrderQueueMessageSyncTransformer : IVirtualQueueMessageTransformer { public async Task> TransformAsync(OrderPlacedQueueMessage message, EnqueueContext? context) { return new( new ExportOrderQueueMessage { Id = message.Id }, new EnqueueContext { ProcessAfter = TimeSpan.FromMinutes(15) } ); } } ``` ## API Virtual queues are extra interesting if you let other systems POST messages into your queues through the API since the virtual queues are also exposed there. This gives you a way to intercept the data sent from the other system and run logic before the message is stored. ## Benefits So when should you use virtual queues? If you have a not-so-large application with just a few developers it might not worth it. But if you have a larger application and maybe even multiple teams it will be worth it. In such a case you should probably create a virtual queue for each queue, and keep the concrete queue message as internal and the virtual queue message as public. That prevents anyone outside to directly manipulate your underlying queue. Something like this: ```cs [VirtualQueue("public_queue")] public class PublicQueueMessage : IQueueMessageWithId { public string? Id { get; set; } } [Queue("internal_queue")] internal class InternalQueueMessage : IQueueMessageWithId { public string? Id { get; set; } } internal class InternalQueueMessageSyncTransformer : IVirtualQueueMessageSyncTransformer { public TransformResult Transform(PublicQueueMessage message, EnqueueContext? context) { return new(new InternalQueueMessage { Id = message.Id }, context); } } ``` Now other .NET projects will only be able to use `IEnqueuer` which gives you full control over your queue. Another option is to start with a public `IQueueMessage` and at a later point convert it from a concrete queue to a virtual queue. Something like this in step 1: ```cs [Queue("internal_queue")] public class PublicQueueMessage : IQueueMessageWithId { public string? Id { get; set; } } ``` And then when you need it change to: ```cs [VirtualQueue("public_queue")] public class PublicQueueMessage : IQueueMessageWithId { public string? Id { get; set; } } [Queue("internal_queue")] internal class InternalQueueMessage : IQueueMessageWithId { public string? Id { get; set; } } ``` There's a couple of downsides with this that you should be aware of: 1. Anyone using the API to enqueue messages needs to update their code to use `public_queue` instead of `internal_queue`. 1. Since we moved the `[Queue("internal_queue")]` to a different type there might still be old JSON messages in that queue that are serialized `PublicQueueMessage`s. ## Delivery and transactions Transformers run in the current scope and can use scoped dependencies. Database/in-memory destinations follow the [Entity Framework unit of work](entity-framework.md) when that integration is enabled. Fan-out does not provide an atomic transaction across remote APIs or external providers; a retry after partial delivery can repeat destinations that already succeeded. Make handlers safe to retry. Routing to another virtual queue is supported, and cycles are rejected. ## Named destinations The typed single-message transformer remains the default. To select a named queue, add `queueName: "destination"` to `TransformResult`. Use `TransformResult.ToMany(...)` for several named destinations. See [named queues](named.md#virtual-entry-points) for registration and examples. A named destination always requires a name, even when it is the only queue for its CLR type. --- # Webhooks for Nexus queues [Webhooks](https://en.wikipedia.org/wiki/Webhook) is a popular way of letting another web application notify your web application that something has happened by posting data to an endpoint in your application. Nexus supports this out-of-the-box by automatically creating an endpoint for each queue in the system that can be used given to another application to send notifications to. The webhook url is: ```text https://url-to-your-nexus-app.com/api/queues/[queue name]/webhook ``` You can also click on the More button in the admin url of your queue to get the exact url. ## Request details Nexus requires the incoming HTTP request to be a POST request and to have a request body in JSON. The body can either be a single message or an array of messages: Either: ```json { "someMessageProperty": "some value" } ``` Or: ```json [ { "someMessageProperty": "message1" }, { "someMessageProperty": "message1" }, { "someMessageProperty": "message1" } ] ``` The JSON body is deserialized into the structure of your queue message and sent to the queue. Which means that any properties in the JSON body that doesn't exist in your message type is discarded, so make sure to include all the properties you need. You can customize the message deserialization by implementing your own `IQueueMessageSerializer` in which you can take the entire original JSON body and store in a separate property on your message. It's also a good idea to use [virtual queues](./virtual.md) for webhooks since it lets you register a single webhook and then pass the webhook notitication to multiple internal queues. --- # Terms of use Nexus is free-of-charge and licensed under the [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0) license. It is however not publically available as the repository is private and the NuGet packages are hosted on a private feed. It may seem counter-intuitive to have an open source licensed code base in a private repository but the reason for it is that we want to know who is using it and be able to help out. It's possible that we at some point will release Nexus completely public. If you want access just hit us an email at [movefaster@commercemind.se](mailto:movefaster@commercemind.se)! The important thing for you to know is that once you've gotten access to the repository and NuGet packages your access to it is protected by the [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0) license. Which means that you will forever have the right to the code and do whatever you want with it as long as you follow the license terms.